Sep 15 10:18:00.025716 ip-10-0-132-90 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 10:18:00.365796 ip-10-0-132-90 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 10:18:00.365796 ip-10-0-132-90 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 10:18:00.365796 ip-10-0-132-90 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 10:18:00.365796 ip-10-0-132-90 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 10:18:00.365796 ip-10-0-132-90 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 10:18:00.365796 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.364703    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 10:18:00.368212 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368195    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 10:18:00.368212 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368212    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368216    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368221    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368224    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368227    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368230    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368233    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368236    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368239    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368242    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368246    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368249    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368252    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368254    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368257    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368259    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368262    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368264    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368268    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368270    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 10:18:00.368278 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368272    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368275    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368278    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368280    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368283    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368286    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368289    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368291    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368294    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368297    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368299    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368302    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368304    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368307    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368309    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368312    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368314    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368317    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368320    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368323    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 10:18:00.368777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368325    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368327    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368330    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368335    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368339    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368341    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368344    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368347    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368350    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368352    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368355    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368358    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368360    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368362    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368369    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368373    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368376    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368379    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368382    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 10:18:00.369258 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368385    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368388    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368391    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368406    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368409    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368411    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368414    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368417    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368420    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368422    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368425    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368430    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368433    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368436    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368439    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368441    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368444    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368447    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368449    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 10:18:00.369745 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368452    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368454    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368457    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368459    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368462    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368464    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368468    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368471    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368474    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368476    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368479    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368482    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368485    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368487    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368490    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368493    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368495    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368498    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368501    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.368503    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 10:18:00.370201 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369234    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369241    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369244    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369248    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369250    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369255    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369258    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369262    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369265    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369268    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369272    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369276    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369279    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369282    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369284    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369287    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369289    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369292    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369294    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 10:18:00.370732 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369297    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369300    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369302    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369305    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369308    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369310    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369313    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369315    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369318    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369320    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369323    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369326    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369328    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369331    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369333    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369336    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369338    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369341    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369343    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369346    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 10:18:00.371204 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369349    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369351    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369353    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369356    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369361    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369364    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369367    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369370    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369373    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369376    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369379    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369382    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369384    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369387    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369390    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369407    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369410    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369413    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369416    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 10:18:00.371701 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369419    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369422    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369425    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369427    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369430    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369433    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369435    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369439    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369443    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369445    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369448    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369452    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369455    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369458    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369461    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369463    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369466    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369468    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369471    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 10:18:00.372161 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369473    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369476    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369478    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369481    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369483    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369486    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369489    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369494    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369497    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369500    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369502    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369505    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369508    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369511    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369513    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369516    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369519    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369521    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369524    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369527    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369529    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 10:18:00.372637 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369532    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369908    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369912    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369915    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369918    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369921    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369924    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369927    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369929    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369932    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369934    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369937    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369939    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369942    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369944    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369947    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369950    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369953    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369957    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369960    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 10:18:00.373139 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369963    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369965    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369968    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369971    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369973    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369976    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369979    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369981    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369984    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369986    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369989    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369992    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369994    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369997    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.369999    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370002    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370005    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370008    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370011    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 10:18:00.373644 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370016    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370019    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370023    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370025    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370028    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370031    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370034    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370037    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370042    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370045    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370048    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370053    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370055    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370058    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370061    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370064    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370066    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370069    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370071    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 10:18:00.374113 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370074    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370076    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370079    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370081    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370084    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370086    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370089    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370091    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370094    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370097    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370099    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370102    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370104    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370107    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370110    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370112    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370115    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370117    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370120    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 10:18:00.374585 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370123    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370125    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370127    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370130    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370132    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370137    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370140    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370142    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370145    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370147    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370150    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370153    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370155    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370158    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370160    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370163    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370165    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370167    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370170    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370172    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 10:18:00.375080 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370175    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370177    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370182    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370256    2584 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370300    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370307    2584 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370313    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370319    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370322    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370327    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370331    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370335    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370338    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370341    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370345    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370348    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370351    2584 flags.go:64] FLAG: --cgroup-root=""
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370355    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370358    2584 flags.go:64] FLAG: --client-ca-file=""
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370361    2584 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370364    2584 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370369    2584 flags.go:64] FLAG: --cluster-domain=""
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370372    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370375    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 10:18:00.375577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370379    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370382    2584 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370386    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370389    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370404    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370408    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370411    2584 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370414    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370418    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370421    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370425    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370429    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370433    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370436    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370439    2584 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370442    2584 flags.go:64] FLAG: --enable-server="true"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370446    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370451    2584 flags.go:64] FLAG: --event-burst="100"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370455    2584 flags.go:64] FLAG: --event-qps="50"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370458    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370461    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370464    2584 flags.go:64] FLAG: --eviction-hard=""
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370468    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370470    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370473    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 10:18:00.376162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370484    2584 flags.go:64] FLAG: --eviction-soft=""
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370488    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370491    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370494    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370497    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370500    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370503    2584 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370506    2584 flags.go:64] FLAG: --feature-gates=""
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370510    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370513    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370516    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370520    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370523    2584 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370526    2584 flags.go:64] FLAG: --help="false"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370529    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370532    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370535    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370539    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370542    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370545    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370549    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370552    2584 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370554    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 10:18:00.376760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370558    2584 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370561    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370565    2584 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370567    2584 flags.go:64] FLAG: --kube-reserved=""
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370570    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370573    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370576    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370579    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370582    2584 flags.go:64] FLAG: --lock-file=""
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370586    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370589    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370592    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370598    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370601    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370604    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370606    2584 flags.go:64] FLAG: --logging-format="text"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370609    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370613    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370615    2584 flags.go:64] FLAG: --manifest-url=""
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370618    2584 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370626    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370629    2584 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370633    2584 flags.go:64] FLAG: --max-pods="110"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370636    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370639    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 10:18:00.377357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370642    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370646    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370649    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370652    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370655    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370663    2584 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370666    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370669    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370672    2584 flags.go:64] FLAG: --pod-cidr=""
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370675    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370680    2584 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370684    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370687    2584 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370690    2584 flags.go:64] FLAG: --port="10250"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370693    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370696    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c7989e8f82852d87"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370699    2584 flags.go:64] FLAG: --qos-reserved=""
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370703    2584 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370707    2584 flags.go:64] FLAG: --register-node="true"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370710    2584 flags.go:64] FLAG: --register-with-taints=""
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370713    2584 flags.go:64] FLAG: --registry-burst="10"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370716    2584 flags.go:64] FLAG: --registry-qps="5"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370719    2584 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370722    2584 flags.go:64] FLAG: --reserved-memory=""
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370725    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 10:18:00.377980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370729    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370732    2584 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370735    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370739    2584 flags.go:64] FLAG: --runonce="false"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370742    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370745    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370748    2584 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370751    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370756    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370759    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370763    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370766    2584 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370769    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370772    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370775    2584 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370777    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370781    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370790    2584 flags.go:64] FLAG: --system-cgroups=""
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370793    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370799    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370802    2584 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370805    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370811    2584 flags.go:64] FLAG: --tls-min-version=""
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370815    2584 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370818    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 10:18:00.378725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370821    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370824    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370827    2584 flags.go:64] FLAG: --v="2"
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370832    2584 flags.go:64] FLAG: --version="false"
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370836    2584 flags.go:64] FLAG: --vmodule=""
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370840    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.370843    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370978    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370987    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370990    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370994    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370997    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.370999    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371002    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371005    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371009    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371012    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371015    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371017    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371020    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371022    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371025    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 10:18:00.379327 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371028    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371031    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371033    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371041    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371045    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371049    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371051    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371054    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371057    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371060    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371062    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371065    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371068    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371071    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371074    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371076    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371079    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371082    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371084    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371087    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 10:18:00.379881 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371089    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371092    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371095    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371097    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371100    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371104    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371107    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371109    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371112    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371115    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371119    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371122    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371125    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371128    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371131    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371134    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371144    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371147    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 10:18:00.380369 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371149    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371152    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371155    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371157    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371160    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371162    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371165    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371167    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371170    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371174    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371177    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371180    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371183    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371186    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371189    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371192    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371195    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371198    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371201    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371205    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 10:18:00.380827 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371208    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371210    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371213    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371216    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371219    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371221    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371224    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371226    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371229    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371231    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371234    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371237    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371240    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371243    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371246    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371248    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371251    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371253    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371256    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371259    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 10:18:00.381336 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371261    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371264    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371267    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371269    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371272    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.371274    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.371776    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.380360    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.380376    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.381012    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.381020    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381086    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381091    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381094    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 10:18:00.381854 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381097    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381101    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381104    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381107    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381109    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381112    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381115    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381120    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381124    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381127    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381130    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381133    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381136    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381139    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381141    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381144    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381146    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381149    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381152    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 10:18:00.382236 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381155    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381157    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381160    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381163    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381165    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381168    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381171    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381179    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381183    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381186    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381188    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381191    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381194    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381196    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381199    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381201    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381210    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381212    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381222    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381226    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 10:18:00.382723 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381229    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381232    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381234    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381237    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381240    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381242    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381245    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381248    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381250    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381253    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381255    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381258    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381261    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381263    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381266    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381268    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381272    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381275    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381277    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381280    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 10:18:00.383203 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381291    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381294    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381298    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381302    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381305    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381308    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381311    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381313    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381316    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381319    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381322    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381324    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381327    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381329    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381333    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381337    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381340    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381343    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381346    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 10:18:00.383708 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381349    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381351    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381354    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381356    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381359    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381362    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381364    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381367    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381370    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381372    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381375    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381378    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381381    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381383    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381410    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381414    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381417    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 10:18:00.384175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381420    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.381426    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381594    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381600    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381603    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381606    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381609    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381612    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381615    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381618    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381621    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381623    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381626    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381629    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381632    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 10:18:00.384673 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381635    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381638    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381641    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381643    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381646    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381648    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381651    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381654    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381656    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381659    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381663    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381668    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381670    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381673    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381676    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381686    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381689    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381692    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381694    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 10:18:00.385085 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381698    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381701    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381704    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381707    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381709    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381712    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381714    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381717    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381720    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381722    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381725    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381727    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381730    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381733    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381735    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381738    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381740    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381743    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381745    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 10:18:00.385579 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381748    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381750    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381753    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381755    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381758    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381760    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381763    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381766    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381768    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381771    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381779    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381782    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381784    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381787    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381789    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381792    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381794    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381797    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381799    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381802    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 10:18:00.386168 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381805    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381808    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381810    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381813    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381816    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381818    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381822    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381825    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381827    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381830    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381833    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381835    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381837    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381840    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381842    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381845    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381847    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381850    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381853    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381855    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 10:18:00.386911 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381858    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381861    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381864    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381872    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381875    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381877    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381880    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:00.381882    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.381887    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.382044    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.384768    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.385539    2584 server.go:1019] "Starting client certificate rotation"
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.385658    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 10:18:00.387742 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.386332    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 10:18:00.388114 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.387809    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 10:18:00.391236 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.390496    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 10:18:00.400239 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.400223    2584 log.go:25] "Validated CRI v1 runtime API"
Sep 15 10:18:00.404925 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.404912    2584 log.go:25] "Validated CRI v1 image API"
Sep 15 10:18:00.405903 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.405887    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 10:18:00.411885 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.411863    2584 fs.go:135] Filesystem UUIDs: map[1236ddd0-d6e4-4ecb-9d6f-ad2721a7d866:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7f5f826f-a434-4573-a699-66934df2e2d0:/dev/nvme0n1p4]
Sep 15 10:18:00.411976 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.411886    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 10:18:00.413774 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.413753    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 10:18:00.416896 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.416780    2584 manager.go:216] Machine: {Timestamp:2026-09-15 10:18:00.415613062 +0000 UTC m=+0.292173444 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099727 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2696c1bae9aa7a3ce838f95e60bfb4 SystemUUID:ec2696c1-bae9-aa7a-3ce8-38f95e60bfb4 BootID:607dd97f-c158-4b8f-aad4-0b4e47353758 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:0c:08:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:0c:08:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:f8:93:d0:c1:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 10:18:00.416896 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.416886    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 10:18:00.417096 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.417047    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 10:18:00.420776 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.420752    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 10:18:00.420916 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.420779    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-90.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 10:18:00.421932 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.421922    2584 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 10:18:00.421966 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.421934    2584 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 10:18:00.421966 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.421943    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 10:18:00.422587 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.422578    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 10:18:00.423389 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.423380    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 10:18:00.427452 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.427441    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 10:18:00.427578 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.427568    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 10:18:00.427615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.427600    2584 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 10:18:00.427615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.427609    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 10:18:00.427673 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.427622    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 10:18:00.427673 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.427632    2584 kubelet.go:398] "Adding apiserver pod source"
Sep 15 10:18:00.427673 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.427641    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 10:18:00.428076 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.428050    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 10:18:00.428136 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.428082    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 10:18:00.430317 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.430288    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 10:18:00.433427 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.433411    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 10:18:00.433523 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.433451    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 10:18:00.435537 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435522    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 10:18:00.435607 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435547    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 10:18:00.435607 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435556    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 10:18:00.435607 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435564    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 10:18:00.435607 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435576    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 10:18:00.435607 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435586    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 10:18:00.435607 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435606    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 10:18:00.435815 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435615    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 10:18:00.435815 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435625    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 10:18:00.435815 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435634    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 10:18:00.435815 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435647    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 10:18:00.435815 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.435655    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 10:18:00.436738 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.436726    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 10:18:00.436808 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.436740    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 10:18:00.438339 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.438277    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 10:18:00.438339 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.438280    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-90.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 10:18:00.440412 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.440379    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l95bz"
Sep 15 10:18:00.440552 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.440539    2584 server.go:1268] "Started kubelet"
Sep 15 10:18:00.440628 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.440608    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 10:18:00.440704 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.440678    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 10:18:00.440757 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.440744    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 10:18:00.441373 ip-10-0-132-90 systemd[1]: Started Kubernetes Kubelet.
Sep 15 10:18:00.443105 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.443069    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 10:18:00.445019 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.444943    2584 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 10:18:00.446958 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.446940    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l95bz"
Sep 15 10:18:00.449427 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.449384    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 10:18:00.449516 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.449437    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-90.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 10:18:00.450790 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.449472    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-90.ec2.internal.18d5773144c51cef  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-90.ec2.internal,UID:ip-10-0-132-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-90.ec2.internal,},FirstTimestamp:2026-09-15 10:18:00.440511727 +0000 UTC m=+0.317072108,LastTimestamp:2026-09-15 10:18:00.440511727 +0000 UTC m=+0.317072108,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-90.ec2.internal,}"
Sep 15 10:18:00.451852 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.451833    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 10:18:00.452277 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.452257    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 10:18:00.452374 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.452347    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:00.453061 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453046    2584 factory.go:55] Registering systemd factory
Sep 15 10:18:00.453061 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453062    2584 factory.go:223] Registration of the systemd container factory successfully
Sep 15 10:18:00.453354 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453326    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 10:18:00.453354 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453334    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 10:18:00.453487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453355    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 10:18:00.453528 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453489    2584 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 10:18:00.453528 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453500    2584 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 10:18:00.453675 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453663    2584 factory.go:153] Registering CRI-O factory
Sep 15 10:18:00.453706 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453682    2584 factory.go:223] Registration of the crio container factory successfully
Sep 15 10:18:00.453746 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453733    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 10:18:00.453784 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453757    2584 factory.go:103] Registering Raw factory
Sep 15 10:18:00.453784 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.453772    2584 manager.go:1196] Started watching for new ooms in manager
Sep 15 10:18:00.454218 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.454191    2584 manager.go:349] Starting recovery of all containers
Sep 15 10:18:00.455477 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.455454    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 10:18:00.459134 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.458262    2584 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-90.ec2.internal\" not found" node="ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.466548 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.466532    2584 manager.go:354] Recovery completed
Sep 15 10:18:00.471874 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.471861    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 10:18:00.474037 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.474017    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 10:18:00.474119 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.474102    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 10:18:00.474119 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.474119    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientPID"
Sep 15 10:18:00.474945 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.474836    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 10:18:00.475015 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.474948    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 10:18:00.475015 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.474964    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 10:18:00.475015 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.474850    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 10:18:00.475862 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.475847    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 10:18:00.475924 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.475900    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 10:18:00.475924 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.475912    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientPID"
Sep 15 10:18:00.477068 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.477056    2584 policy_none.go:49] "None policy: Start"
Sep 15 10:18:00.477113 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.477073    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 10:18:00.477113 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.477084    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 10:18:00.479040 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.479027    2584 policy_none.go:47] "Start"
Sep 15 10:18:00.525175 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.518078    2584 manager.go:337] "Starting Device Plugin manager"
Sep 15 10:18:00.525175 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.518122    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 10:18:00.525175 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.518131    2584 server.go:87] "Starting device plugin registration server"
Sep 15 10:18:00.525175 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.518582    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 10:18:00.525175 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.518630    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 10:18:00.525175 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.518708    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 10:18:00.525175 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.518842    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 10:18:00.525175 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.518854    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 10:18:00.525175 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.519819    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 10:18:00.525175 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.519864    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:00.594785 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.594749    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 10:18:00.595962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.595947    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 10:18:00.596044 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.596008    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 10:18:00.596044 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.596039    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 10:18:00.596153 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.596136    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 10:18:00.598122 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.598102    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 10:18:00.618933 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.618913    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 10:18:00.619945 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.619929    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 10:18:00.620052 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.619960    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 10:18:00.620052 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.619975    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientPID"
Sep 15 10:18:00.620052 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.620005    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.626748 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.626728    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.626748 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.626749    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-90.ec2.internal\": node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:00.642070 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.642053    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:00.696660 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.696543    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-90.ec2.internal"]
Sep 15 10:18:00.696660 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.696639    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 10:18:00.697654 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.697636    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 10:18:00.697743 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.697666    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 10:18:00.697743 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.697683    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientPID"
Sep 15 10:18:00.700067 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.700054    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 10:18:00.700205 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.700190    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.700280 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.700229    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 10:18:00.700856 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.700839    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 10:18:00.700930 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.700865    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 10:18:00.700930 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.700876    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientPID"
Sep 15 10:18:00.700930 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.700847    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 10:18:00.701040 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.700941    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 10:18:00.701040 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.700952    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientPID"
Sep 15 10:18:00.703327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.703312    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.703378 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.703344    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 10:18:00.704087 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.704068    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 10:18:00.704087 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.704092    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 10:18:00.704221 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.704104    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeHasSufficientPID"
Sep 15 10:18:00.727152 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.727129    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-90.ec2.internal\" not found" node="ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.731703 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.731688    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-90.ec2.internal\" not found" node="ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.742749 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.742729    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:00.754697 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.754669    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3c08f4f6e555d16caa02f9df8c01702-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal\" (UID: \"d3c08f4f6e555d16caa02f9df8c01702\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.754791 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.754703    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3c08f4f6e555d16caa02f9df8c01702-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal\" (UID: \"d3c08f4f6e555d16caa02f9df8c01702\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.754791 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.754724    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18620f5e88bb272825dcdf930e2ff0ad-config\") pod \"kube-apiserver-proxy-ip-10-0-132-90.ec2.internal\" (UID: \"18620f5e88bb272825dcdf930e2ff0ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.843492 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.843454    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:00.854838 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.854805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3c08f4f6e555d16caa02f9df8c01702-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal\" (UID: \"d3c08f4f6e555d16caa02f9df8c01702\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.854911 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.854842    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3c08f4f6e555d16caa02f9df8c01702-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal\" (UID: \"d3c08f4f6e555d16caa02f9df8c01702\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.854911 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.854865    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18620f5e88bb272825dcdf930e2ff0ad-config\") pod \"kube-apiserver-proxy-ip-10-0-132-90.ec2.internal\" (UID: \"18620f5e88bb272825dcdf930e2ff0ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.854984 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.854912    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18620f5e88bb272825dcdf930e2ff0ad-config\") pod \"kube-apiserver-proxy-ip-10-0-132-90.ec2.internal\" (UID: \"18620f5e88bb272825dcdf930e2ff0ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.854984 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.854930    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3c08f4f6e555d16caa02f9df8c01702-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal\" (UID: \"d3c08f4f6e555d16caa02f9df8c01702\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.854984 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:00.854930    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3c08f4f6e555d16caa02f9df8c01702-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal\" (UID: \"d3c08f4f6e555d16caa02f9df8c01702\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:00.944314 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:00.944232    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:01.029773 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:01.029710    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:01.034888 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:01.034665    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:01.044379 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:01.044358    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:01.144979 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:01.144928    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:02.427920 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:02.427888    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:02.528511 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:02.528438    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:02.567987 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:02.567962    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 10:18:02.568450 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:02.568112    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 10:18:02.568450 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:02.568144    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 10:18:02.629485 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:02.629453    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-90.ec2.internal\" not found"
Sep 15 10:18:02.635388 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:02.635353    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 10:13:01 +0000 UTC" deadline="2028-03-08 07:42:50.040887043 +0000 UTC"
Sep 15 10:18:02.635439 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:02.635389    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12957h24m47.405501204s"
Sep 15 10:18:02.702595 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:02.702561    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 10:18:02.735554 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:02.735529    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:02.745410 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:02.745388    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 10:18:02.747154 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:02.747137    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-90.ec2.internal"
Sep 15 10:18:02.755326 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:02.755294    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 10:18:03.005656 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.005577    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 10:18:03.132047 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:03.132005    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18620f5e88bb272825dcdf930e2ff0ad.slice/crio-beb0ac27f04dbeca7c9d79a6296785d563c515ef8addb1df7f455a7a0a70e6cf WatchSource:0}: Error finding container beb0ac27f04dbeca7c9d79a6296785d563c515ef8addb1df7f455a7a0a70e6cf: Status 404 returned error can't find the container with id beb0ac27f04dbeca7c9d79a6296785d563c515ef8addb1df7f455a7a0a70e6cf
Sep 15 10:18:03.132444 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:03.132425    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c08f4f6e555d16caa02f9df8c01702.slice/crio-3bf5c7ee4d9cb289efe0437ca170fafcf751278545438dc6974f46a43b82655c WatchSource:0}: Error finding container 3bf5c7ee4d9cb289efe0437ca170fafcf751278545438dc6974f46a43b82655c: Status 404 returned error can't find the container with id 3bf5c7ee4d9cb289efe0437ca170fafcf751278545438dc6974f46a43b82655c
Sep 15 10:18:03.136365 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.136346    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 10:18:03.610837 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.610793    2584 apiserver.go:52] "Watching apiserver"
Sep 15 10:18:03.619366 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.619339    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 10:18:03.619776 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.619747    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-q4zj5","openshift-multus/multus-zf2rw","openshift-network-diagnostics/network-check-target-dszpz","openshift-ovn-kubernetes/ovnkube-node-nnbj8","kube-system/konnectivity-agent-n8bbj","openshift-cluster-node-tuning-operator/tuned-cdxsk","openshift-image-registry/node-ca-7vfnf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal","openshift-multus/multus-additional-cni-plugins-bgms7","openshift-multus/network-metrics-daemon-bftt2","openshift-network-operator/iptables-alerter-xhsgp","kube-system/global-pull-secret-syncer-txhqg","kube-system/kube-apiserver-proxy-ip-10-0-132-90.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"]
Sep 15 10:18:03.622283 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.622257    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7vfnf"
Sep 15 10:18:03.624505 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.624482    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xhsgp"
Sep 15 10:18:03.624616 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.624539    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 10:18:03.624616 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.624557    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 10:18:03.624861 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.624834    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 10:18:03.624980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.624877    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nvkmf\""
Sep 15 10:18:03.626615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.626588    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 10:18:03.626615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.626609    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ncv85\""
Sep 15 10:18:03.626905 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.626889    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 10:18:03.627023 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.627002    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:03.627101 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:03.627070    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:03.627222 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.627201    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 10:18:03.629812 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.629781    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.632150 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.632131    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 10:18:03.632342 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.632327    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 10:18:03.632573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.632551    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 10:18:03.632675 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.632591    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 10:18:03.632735 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.632559    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:03.633021 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.632831    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 10:18:03.633021 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.632840    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 10:18:03.633021 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.632918    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pccfn\""
Sep 15 10:18:03.634582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.634565    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.634818 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.634783    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fb6pf\""
Sep 15 10:18:03.635034 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.635017    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 10:18:03.635126 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.635101    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 10:18:03.636689 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.635919    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 10:18:03.637360 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.637338    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 10:18:03.637614 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.637597    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 10:18:03.637911 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.637878    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c7bqh\""
Sep 15 10:18:03.638000 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.637983    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.640908 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.640876    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.644882 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.644858    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmx9\" (UniqueName: \"kubernetes.io/projected/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-kube-api-access-7kmx9\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.644975 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.644902    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-kubernetes\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.644975 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.644932    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-etc-selinux\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.644975 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.644972    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8vx7\" (UniqueName: \"kubernetes.io/projected/f92feea8-a50a-4580-bc28-281661e98802-kube-api-access-q8vx7\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.645141 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.644999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b958d69-b275-4c2f-8f0a-31ba329cdc25-tmp\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.645141 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645022    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-kubelet\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.645141 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645049    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-run-systemd\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.645141 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645076    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.645141 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645107    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0582a9b-c2c1-437e-99a6-0a907acb9b35-konnectivity-ca\") pod \"konnectivity-agent-n8bbj\" (UID: \"a0582a9b-c2c1-437e-99a6-0a907acb9b35\") " pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:03.645141 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645136    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-etc-openvswitch\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.645424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645164    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-run-ovn\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.645424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645187    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-ovnkube-config\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.645424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645214    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-lib-modules\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.645424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645243    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.645424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645281    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-modprobe-d\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.645424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645303    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-socket-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.645424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645341    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72c44564-2d7f-47fd-8567-d8793ed885bd-host\") pod \"node-ca-7vfnf\" (UID: \"72c44564-2d7f-47fd-8567-d8793ed885bd\") " pod="openshift-image-registry/node-ca-7vfnf"
Sep 15 10:18:03.645424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645372    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-run-openvswitch\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.645424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645399    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-cni-bin\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.645872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645427    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33162a7a-bbbc-475d-b186-63e2b94656fe-cnibin\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.645872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645465    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33162a7a-bbbc-475d-b186-63e2b94656fe-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.645872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645493    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-sysconfig\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.645872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645527    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-slash\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.645872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645567    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-ovn-node-metrics-cert\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.645872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645595    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-host\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.645872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645623    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qsr\" (UniqueName: \"kubernetes.io/projected/8b958d69-b275-4c2f-8f0a-31ba329cdc25-kube-api-access-s7qsr\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.645872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645647    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33162a7a-bbbc-475d-b186-63e2b94656fe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.645872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645679    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/33162a7a-bbbc-475d-b186-63e2b94656fe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.645872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645709    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5bbf8a3-ce12-4124-9077-e892fef0316f-host-slash\") pod \"iptables-alerter-xhsgp\" (UID: \"f5bbf8a3-ce12-4124-9077-e892fef0316f\") " pod="openshift-network-operator/iptables-alerter-xhsgp"
Sep 15 10:18:03.645872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645809    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-node-log\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.646299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.645889    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33162a7a-bbbc-475d-b186-63e2b94656fe-system-cni-dir\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.646299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646011    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-ovnkube-script-lib\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.646299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646040    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-sysctl-conf\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.646299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646064    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-device-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.646299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646142    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5bbf8a3-ce12-4124-9077-e892fef0316f-iptables-alerter-script\") pod \"iptables-alerter-xhsgp\" (UID: \"f5bbf8a3-ce12-4124-9077-e892fef0316f\") " pod="openshift-network-operator/iptables-alerter-xhsgp"
Sep 15 10:18:03.646299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646172    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-cni-netd\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.646299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646215    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-systemd\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.646299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646268    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-tuned\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.646299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646299    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqsfz\" (UniqueName: \"kubernetes.io/projected/33162a7a-bbbc-475d-b186-63e2b94656fe-kube-api-access-mqsfz\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.646772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646324    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72c44564-2d7f-47fd-8567-d8793ed885bd-serviceca\") pod \"node-ca-7vfnf\" (UID: \"72c44564-2d7f-47fd-8567-d8793ed885bd\") " pod="openshift-image-registry/node-ca-7vfnf"
Sep 15 10:18:03.646772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646488    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-var-lib-openvswitch\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.646772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-env-overrides\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.646772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646548    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-sysctl-d\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.646772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646564    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 10:18:03.646772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646577    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.646772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646637    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmznj\" (UniqueName: \"kubernetes.io/projected/72c44564-2d7f-47fd-8567-d8793ed885bd-kube-api-access-kmznj\") pod \"node-ca-7vfnf\" (UID: \"72c44564-2d7f-47fd-8567-d8793ed885bd\") " pod="openshift-image-registry/node-ca-7vfnf"
Sep 15 10:18:03.646772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646665    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-log-socket\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.646772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646691    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-systemd-units\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.646772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646703    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9c9rd\""
Sep 15 10:18:03.646772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646738    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-run-netns\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.647348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646778    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0582a9b-c2c1-437e-99a6-0a907acb9b35-agent-certs\") pod \"konnectivity-agent-n8bbj\" (UID: \"a0582a9b-c2c1-437e-99a6-0a907acb9b35\") " pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:03.647348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646837    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-var-lib-kubelet\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.647348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646862    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 10:18:03.647348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646891    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-registration-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.647348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646919    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-sys-fs\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.647348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.646518    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 10:18:03.647348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.647177    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.647348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.647193    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 10:18:03.647348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.647253    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 10:18:03.647348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.647323    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 10:18:03.647967 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.647379    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5tv4m\""
Sep 15 10:18:03.647967 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.647570    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-run\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.647967 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.647626    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/33162a7a-bbbc-475d-b186-63e2b94656fe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.647967 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.647688    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rklmm\" (UniqueName: \"kubernetes.io/projected/f5bbf8a3-ce12-4124-9077-e892fef0316f-kube-api-access-rklmm\") pod \"iptables-alerter-xhsgp\" (UID: \"f5bbf8a3-ce12-4124-9077-e892fef0316f\") " pod="openshift-network-operator/iptables-alerter-xhsgp"
Sep 15 10:18:03.647967 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.647743    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 10:18:03.647967 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.647767    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9kq6\" (UniqueName: \"kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6\") pod \"network-check-target-dszpz\" (UID: \"aab62d15-9d50-4842-9f39-0ed41176b31a\") " pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:03.647967 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.647884    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 10:18:03.648309 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.648026    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-sys\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.648958 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.648938    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:03.649042 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:03.649015    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:03.649447 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.649427    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 10:18:03.649553 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.649536    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ngz5l\""
Sep 15 10:18:03.652007 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.651983    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q4zj5"
Sep 15 10:18:03.652968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.652948    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 10:18:03.654110 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.654087    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k8jwj\""
Sep 15 10:18:03.654202 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.654143    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 10:18:03.654202 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.654192    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 10:18:03.654360 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.654340    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:03.654421 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:03.654404    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:03.669021 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.669000    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x7ckc"
Sep 15 10:18:03.678975 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.678952    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x7ckc"
Sep 15 10:18:03.736627 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.736596    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 10:18:03.748684 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.748651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-host\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.748845 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.748697    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qsr\" (UniqueName: \"kubernetes.io/projected/8b958d69-b275-4c2f-8f0a-31ba329cdc25-kube-api-access-s7qsr\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.748845 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.748726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33162a7a-bbbc-475d-b186-63e2b94656fe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.748845 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.748753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/33162a7a-bbbc-475d-b186-63e2b94656fe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.748845 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.748743    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-host\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.749069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.748940    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33162a7a-bbbc-475d-b186-63e2b94656fe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.749069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.748997    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1e9ea1a-0387-4294-a760-df3980219768-hosts-file\") pod \"node-resolver-q4zj5\" (UID: \"b1e9ea1a-0387-4294-a760-df3980219768\") " pod="openshift-dns/node-resolver-q4zj5"
Sep 15 10:18:03.749069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749027    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5bbf8a3-ce12-4124-9077-e892fef0316f-host-slash\") pod \"iptables-alerter-xhsgp\" (UID: \"f5bbf8a3-ce12-4124-9077-e892fef0316f\") " pod="openshift-network-operator/iptables-alerter-xhsgp"
Sep 15 10:18:03.749069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749064    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-node-log\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.749307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749110    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-node-log\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.749307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749141    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33162a7a-bbbc-475d-b186-63e2b94656fe-system-cni-dir\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.749307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749148    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5bbf8a3-ce12-4124-9077-e892fef0316f-host-slash\") pod \"iptables-alerter-xhsgp\" (UID: \"f5bbf8a3-ce12-4124-9077-e892fef0316f\") " pod="openshift-network-operator/iptables-alerter-xhsgp"
Sep 15 10:18:03.749307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749177    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pv4k\" (UniqueName: \"kubernetes.io/projected/b1e9ea1a-0387-4294-a760-df3980219768-kube-api-access-5pv4k\") pod \"node-resolver-q4zj5\" (UID: \"b1e9ea1a-0387-4294-a760-df3980219768\") " pod="openshift-dns/node-resolver-q4zj5"
Sep 15 10:18:03.749307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/992c0c97-0af1-4197-82ca-e4c193a935f6-kubelet-config\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:03.749307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749222    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8246177-ac46-487b-9853-e979365daa1f-multus-daemon-config\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.749307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749242    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-etc-kubernetes\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.749307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749263    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-ovnkube-script-lib\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.749307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749278    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33162a7a-bbbc-475d-b186-63e2b94656fe-system-cni-dir\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.749307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749287    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-sysctl-conf\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.749307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749310    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-device-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-multus-conf-dir\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749358    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5bbf8a3-ce12-4124-9077-e892fef0316f-iptables-alerter-script\") pod \"iptables-alerter-xhsgp\" (UID: \"f5bbf8a3-ce12-4124-9077-e892fef0316f\") " pod="openshift-network-operator/iptables-alerter-xhsgp"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749363    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/33162a7a-bbbc-475d-b186-63e2b94656fe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749398    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-cni-netd\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-systemd\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749439    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-tuned\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749466    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-device-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749467    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-sysctl-conf\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749512    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-cni-netd\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749538    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-systemd\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqsfz\" (UniqueName: \"kubernetes.io/projected/33162a7a-bbbc-475d-b186-63e2b94656fe-kube-api-access-mqsfz\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749615    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8246177-ac46-487b-9853-e979365daa1f-cni-binary-copy\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749636    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72c44564-2d7f-47fd-8567-d8793ed885bd-serviceca\") pod \"node-ca-7vfnf\" (UID: \"72c44564-2d7f-47fd-8567-d8793ed885bd\") " pod="openshift-image-registry/node-ca-7vfnf"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749672    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-var-lib-openvswitch\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749709    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-env-overrides\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749745    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-sysctl-d\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.749847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749792    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gdxl\" (UniqueName: \"kubernetes.io/projected/6992d04d-f1f5-4521-a0da-27555d562676-kube-api-access-4gdxl\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749836    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmznj\" (UniqueName: \"kubernetes.io/projected/72c44564-2d7f-47fd-8567-d8793ed885bd-kube-api-access-kmznj\") pod \"node-ca-7vfnf\" (UID: \"72c44564-2d7f-47fd-8567-d8793ed885bd\") " pod="openshift-image-registry/node-ca-7vfnf"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749835    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749900    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-log-socket\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749978    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5bbf8a3-ce12-4124-9077-e892fef0316f-iptables-alerter-script\") pod \"iptables-alerter-xhsgp\" (UID: \"f5bbf8a3-ce12-4124-9077-e892fef0316f\") " pod="openshift-network-operator/iptables-alerter-xhsgp"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750049    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-ovnkube-script-lib\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750217    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-sysctl-d\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750271    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750320    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-var-lib-openvswitch\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.749864    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-log-socket\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-env-overrides\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750429    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-system-cni-dir\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750459    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-run-multus-certs\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750511    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-systemd-units\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750556    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-run-netns\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750579    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0582a9b-c2c1-437e-99a6-0a907acb9b35-agent-certs\") pod \"konnectivity-agent-n8bbj\" (UID: \"a0582a9b-c2c1-437e-99a6-0a907acb9b35\") " pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750577    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72c44564-2d7f-47fd-8567-d8793ed885bd-serviceca\") pod \"node-ca-7vfnf\" (UID: \"72c44564-2d7f-47fd-8567-d8793ed885bd\") " pod="openshift-image-registry/node-ca-7vfnf"
Sep 15 10:18:03.750661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750600    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-systemd-units\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750620    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-run-netns\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-var-lib-kubelet\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750680    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-registration-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-sys-fs\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750727    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e9ea1a-0387-4294-a760-df3980219768-tmp-dir\") pod \"node-resolver-q4zj5\" (UID: \"b1e9ea1a-0387-4294-a760-df3980219768\") " pod="openshift-dns/node-resolver-q4zj5"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750752    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-run\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750776    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/33162a7a-bbbc-475d-b186-63e2b94656fe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750819    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/992c0c97-0af1-4197-82ca-e4c193a935f6-dbus\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750844    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-cnibin\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750870    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rklmm\" (UniqueName: \"kubernetes.io/projected/f5bbf8a3-ce12-4124-9077-e892fef0316f-kube-api-access-rklmm\") pod \"iptables-alerter-xhsgp\" (UID: \"f5bbf8a3-ce12-4124-9077-e892fef0316f\") " pod="openshift-network-operator/iptables-alerter-xhsgp"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750911    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kq6\" (UniqueName: \"kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6\") pod \"network-check-target-dszpz\" (UID: \"aab62d15-9d50-4842-9f39-0ed41176b31a\") " pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750934    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-sys\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.750956    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751006    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-run-netns\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-var-lib-cni-multus\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmx9\" (UniqueName: \"kubernetes.io/projected/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-kube-api-access-7kmx9\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.751570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-kubernetes\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751112    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-etc-selinux\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8vx7\" (UniqueName: \"kubernetes.io/projected/f92feea8-a50a-4580-bc28-281661e98802-kube-api-access-q8vx7\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751160    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-multus-cni-dir\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751199    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-var-lib-cni-bin\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751219    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b958d69-b275-4c2f-8f0a-31ba329cdc25-tmp\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751239    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-kubelet\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751294    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-run-systemd\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751322    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751344    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0582a9b-c2c1-437e-99a6-0a907acb9b35-konnectivity-ca\") pod \"konnectivity-agent-n8bbj\" (UID: \"a0582a9b-c2c1-437e-99a6-0a907acb9b35\") " pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751369    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcxln\" (UniqueName: \"kubernetes.io/projected/c8246177-ac46-487b-9853-e979365daa1f-kube-api-access-xcxln\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751389    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751411    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-etc-openvswitch\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751431    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-run-ovn\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-ovnkube-config\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751471    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-lib-modules\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751495    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.752424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-modprobe-d\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751558    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-socket-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751601    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-hostroot\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751639    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72c44564-2d7f-47fd-8567-d8793ed885bd-host\") pod \"node-ca-7vfnf\" (UID: \"72c44564-2d7f-47fd-8567-d8793ed885bd\") " pod="openshift-image-registry/node-ca-7vfnf"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751664    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-run-openvswitch\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-cni-bin\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33162a7a-bbbc-475d-b186-63e2b94656fe-cnibin\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751747    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33162a7a-bbbc-475d-b186-63e2b94656fe-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751773    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-sysconfig\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751815    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-multus-socket-dir-parent\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751843    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-run-k8s-cni-cncf-io\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751868    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-var-lib-kubelet\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-slash\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751917    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-ovn-node-metrics-cert\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.751938    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-sys-fs\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752037    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-etc-selinux\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752067    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-kubernetes\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.753306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752069    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/33162a7a-bbbc-475d-b186-63e2b94656fe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-run\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752131    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752163    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-sysconfig\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752225    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-run-openvswitch\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752234    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33162a7a-bbbc-475d-b186-63e2b94656fe-cnibin\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752255    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-modprobe-d\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-run-ovn\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752318    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-slash\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752331    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-etc-openvswitch\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752369    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-sys\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752379    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-registration-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752402    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-cni-bin\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752456    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-host-kubelet\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752531    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-lib-modules\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752574    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b958d69-b275-4c2f-8f0a-31ba329cdc25-var-lib-kubelet\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752617    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-run-systemd\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.754120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752671    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72c44564-2d7f-47fd-8567-d8793ed885bd-host\") pod \"node-ca-7vfnf\" (UID: \"72c44564-2d7f-47fd-8567-d8793ed885bd\") " pod="openshift-image-registry/node-ca-7vfnf"
Sep 15 10:18:03.754990 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752789    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f92feea8-a50a-4580-bc28-281661e98802-socket-dir\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.754990 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752827    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0582a9b-c2c1-437e-99a6-0a907acb9b35-konnectivity-ca\") pod \"konnectivity-agent-n8bbj\" (UID: \"a0582a9b-c2c1-437e-99a6-0a907acb9b35\") " pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:03.754990 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.752878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-ovnkube-config\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.754990 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.753240    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33162a7a-bbbc-475d-b186-63e2b94656fe-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.754990 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.754832    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b958d69-b275-4c2f-8f0a-31ba329cdc25-etc-tuned\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.755718 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.755696    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0582a9b-c2c1-437e-99a6-0a907acb9b35-agent-certs\") pod \"konnectivity-agent-n8bbj\" (UID: \"a0582a9b-c2c1-437e-99a6-0a907acb9b35\") " pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:03.756677 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.756651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b958d69-b275-4c2f-8f0a-31ba329cdc25-tmp\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.757047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.757024    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-ovn-node-metrics-cert\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.757526 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.757261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qsr\" (UniqueName: \"kubernetes.io/projected/8b958d69-b275-4c2f-8f0a-31ba329cdc25-kube-api-access-s7qsr\") pod \"tuned-cdxsk\" (UID: \"8b958d69-b275-4c2f-8f0a-31ba329cdc25\") " pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.759520 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:03.759495    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 10:18:03.759629 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:03.759528    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 10:18:03.759629 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:03.759541    2584 projected.go:196] Error preparing data for projected volume kube-api-access-x9kq6 for pod openshift-network-diagnostics/network-check-target-dszpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:03.759968 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:03.759947    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6 podName:aab62d15-9d50-4842-9f39-0ed41176b31a nodeName:}" failed. No retries permitted until 2026-09-15 10:18:04.259913379 +0000 UTC m=+2.954069053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x9kq6" (UniqueName: "kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6") pod "network-check-target-dszpz" (UID: "aab62d15-9d50-4842-9f39-0ed41176b31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:03.762286 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.762261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rklmm\" (UniqueName: \"kubernetes.io/projected/f5bbf8a3-ce12-4124-9077-e892fef0316f-kube-api-access-rklmm\") pod \"iptables-alerter-xhsgp\" (UID: \"f5bbf8a3-ce12-4124-9077-e892fef0316f\") " pod="openshift-network-operator/iptables-alerter-xhsgp"
Sep 15 10:18:03.762362 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.762318    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqsfz\" (UniqueName: \"kubernetes.io/projected/33162a7a-bbbc-475d-b186-63e2b94656fe-kube-api-access-mqsfz\") pod \"multus-additional-cni-plugins-bgms7\" (UID: \"33162a7a-bbbc-475d-b186-63e2b94656fe\") " pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.762858 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.762835    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmznj\" (UniqueName: \"kubernetes.io/projected/72c44564-2d7f-47fd-8567-d8793ed885bd-kube-api-access-kmznj\") pod \"node-ca-7vfnf\" (UID: \"72c44564-2d7f-47fd-8567-d8793ed885bd\") " pod="openshift-image-registry/node-ca-7vfnf"
Sep 15 10:18:03.765154 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.763421    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8vx7\" (UniqueName: \"kubernetes.io/projected/f92feea8-a50a-4580-bc28-281661e98802-kube-api-access-q8vx7\") pod \"aws-ebs-csi-driver-node-2xq8z\" (UID: \"f92feea8-a50a-4580-bc28-281661e98802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.768343 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.768320    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmx9\" (UniqueName: \"kubernetes.io/projected/b996eb8a-182a-48ab-9aa6-6be28e14c8ac-kube-api-access-7kmx9\") pod \"ovnkube-node-nnbj8\" (UID: \"b996eb8a-182a-48ab-9aa6-6be28e14c8ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.785113 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.785055    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal" event={"ID":"d3c08f4f6e555d16caa02f9df8c01702","Type":"ContainerStarted","Data":"3bf5c7ee4d9cb289efe0437ca170fafcf751278545438dc6974f46a43b82655c"}
Sep 15 10:18:03.787134 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.787099    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-90.ec2.internal" event={"ID":"18620f5e88bb272825dcdf930e2ff0ad","Type":"ContainerStarted","Data":"beb0ac27f04dbeca7c9d79a6296785d563c515ef8addb1df7f455a7a0a70e6cf"}
Sep 15 10:18:03.852375 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852339    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-multus-cni-dir\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852552 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852386    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-var-lib-cni-bin\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852552 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcxln\" (UniqueName: \"kubernetes.io/projected/c8246177-ac46-487b-9853-e979365daa1f-kube-api-access-xcxln\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852552 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852436    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:03.852552 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852460    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-hostroot\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852552 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-multus-socket-dir-parent\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852552 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852509    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-run-k8s-cni-cncf-io\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852552 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852530    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-var-lib-kubelet\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852557    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1e9ea1a-0387-4294-a760-df3980219768-hosts-file\") pod \"node-resolver-q4zj5\" (UID: \"b1e9ea1a-0387-4294-a760-df3980219768\") " pod="openshift-dns/node-resolver-q4zj5"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852580    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pv4k\" (UniqueName: \"kubernetes.io/projected/b1e9ea1a-0387-4294-a760-df3980219768-kube-api-access-5pv4k\") pod \"node-resolver-q4zj5\" (UID: \"b1e9ea1a-0387-4294-a760-df3980219768\") " pod="openshift-dns/node-resolver-q4zj5"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852601    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/992c0c97-0af1-4197-82ca-e4c193a935f6-kubelet-config\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8246177-ac46-487b-9853-e979365daa1f-multus-daemon-config\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852643    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-etc-kubernetes\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852667    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-multus-conf-dir\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852693    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8246177-ac46-487b-9853-e979365daa1f-cni-binary-copy\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852755    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gdxl\" (UniqueName: \"kubernetes.io/projected/6992d04d-f1f5-4521-a0da-27555d562676-kube-api-access-4gdxl\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-system-cni-dir\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852824    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-run-multus-certs\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852850    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-multus-cni-dir\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e9ea1a-0387-4294-a760-df3980219768-tmp-dir\") pod \"node-resolver-q4zj5\" (UID: \"b1e9ea1a-0387-4294-a760-df3980219768\") " pod="openshift-dns/node-resolver-q4zj5"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852904    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-var-lib-cni-bin\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852916    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/992c0c97-0af1-4197-82ca-e4c193a935f6-dbus\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-cnibin\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.852961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852975    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:03.853487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.852996    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-run-netns\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.853487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.853017    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-var-lib-cni-multus\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.853487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.853110    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-var-lib-cni-multus\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.853487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.853174    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1e9ea1a-0387-4294-a760-df3980219768-hosts-file\") pod \"node-resolver-q4zj5\" (UID: \"b1e9ea1a-0387-4294-a760-df3980219768\") " pod="openshift-dns/node-resolver-q4zj5"
Sep 15 10:18:03.853487 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:03.853216    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:03.853487 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:03.853272    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs podName:6992d04d-f1f5-4521-a0da-27555d562676 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:04.353255442 +0000 UTC m=+3.047411114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs") pod "network-metrics-daemon-bftt2" (UID: "6992d04d-f1f5-4521-a0da-27555d562676") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:03.853487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.853398    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/992c0c97-0af1-4197-82ca-e4c193a935f6-kubelet-config\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:03.853487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.853420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-hostroot\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.853487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.853470    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-run-k8s-cni-cncf-io\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.853792 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.853525    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-system-cni-dir\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.853792 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.853560    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-etc-kubernetes\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.853792 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.853600    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-multus-conf-dir\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.854011 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.853990    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8246177-ac46-487b-9853-e979365daa1f-multus-daemon-config\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.854047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.854028    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8246177-ac46-487b-9853-e979365daa1f-cni-binary-copy\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.854080 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.854064    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-multus-socket-dir-parent\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.854130 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.854106    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-run-multus-certs\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.854290 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.854274    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-var-lib-kubelet\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.854349 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.854336    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-cnibin\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.854404 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.854388    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e9ea1a-0387-4294-a760-df3980219768-tmp-dir\") pod \"node-resolver-q4zj5\" (UID: \"b1e9ea1a-0387-4294-a760-df3980219768\") " pod="openshift-dns/node-resolver-q4zj5"
Sep 15 10:18:03.854439 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.854396    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/992c0c97-0af1-4197-82ca-e4c193a935f6-dbus\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:03.854492 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.854460    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8246177-ac46-487b-9853-e979365daa1f-host-run-netns\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.854492 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:03.854482    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:03.854620 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:03.854533    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret podName:992c0c97-0af1-4197-82ca-e4c193a935f6 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:04.354517957 +0000 UTC m=+3.048673844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret") pod "global-pull-secret-syncer-txhqg" (UID: "992c0c97-0af1-4197-82ca-e4c193a935f6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:03.862766 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.862707    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcxln\" (UniqueName: \"kubernetes.io/projected/c8246177-ac46-487b-9853-e979365daa1f-kube-api-access-xcxln\") pod \"multus-zf2rw\" (UID: \"c8246177-ac46-487b-9853-e979365daa1f\") " pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:03.864532 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.864511    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pv4k\" (UniqueName: \"kubernetes.io/projected/b1e9ea1a-0387-4294-a760-df3980219768-kube-api-access-5pv4k\") pod \"node-resolver-q4zj5\" (UID: \"b1e9ea1a-0387-4294-a760-df3980219768\") " pod="openshift-dns/node-resolver-q4zj5"
Sep 15 10:18:03.864676 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.864652    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gdxl\" (UniqueName: \"kubernetes.io/projected/6992d04d-f1f5-4521-a0da-27555d562676-kube-api-access-4gdxl\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:03.930710 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.930673    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 10:18:03.934773 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.934749    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7vfnf"
Sep 15 10:18:03.944903 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.944877    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xhsgp"
Sep 15 10:18:03.949319 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:03.949284    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c44564_2d7f_47fd_8567_d8793ed885bd.slice/crio-d1a952e1ad923ab84da5813f61b1e99e62b05b1a95d728cb67fa3981a8dcc063 WatchSource:0}: Error finding container d1a952e1ad923ab84da5813f61b1e99e62b05b1a95d728cb67fa3981a8dcc063: Status 404 returned error can't find the container with id d1a952e1ad923ab84da5813f61b1e99e62b05b1a95d728cb67fa3981a8dcc063
Sep 15 10:18:03.955761 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:03.955737    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bbf8a3_ce12_4124_9077_e892fef0316f.slice/crio-85b959c6409404019fff9d5cdea29e56c6da39ccdb4d9fa29f5cf62f857cd27b WatchSource:0}: Error finding container 85b959c6409404019fff9d5cdea29e56c6da39ccdb4d9fa29f5cf62f857cd27b: Status 404 returned error can't find the container with id 85b959c6409404019fff9d5cdea29e56c6da39ccdb4d9fa29f5cf62f857cd27b
Sep 15 10:18:03.957138 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.956972    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:03.963163 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.963142    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:03.970512 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:03.970488    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb996eb8a_182a_48ab_9aa6_6be28e14c8ac.slice/crio-8a3a05021f1cd1f32db426af294d9156d142ab3c215dbe1521167a21471be195 WatchSource:0}: Error finding container 8a3a05021f1cd1f32db426af294d9156d142ab3c215dbe1521167a21471be195: Status 404 returned error can't find the container with id 8a3a05021f1cd1f32db426af294d9156d142ab3c215dbe1521167a21471be195
Sep 15 10:18:03.971639 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.971613    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cdxsk"
Sep 15 10:18:03.978394 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.978356    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgms7"
Sep 15 10:18:03.982485 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:03.982465    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b958d69_b275_4c2f_8f0a_31ba329cdc25.slice/crio-b809c12bd9988e5376d9bbe563a4f9fe4bfca669495f9b9df5a2fcc56777a486 WatchSource:0}: Error finding container b809c12bd9988e5376d9bbe563a4f9fe4bfca669495f9b9df5a2fcc56777a486: Status 404 returned error can't find the container with id b809c12bd9988e5376d9bbe563a4f9fe4bfca669495f9b9df5a2fcc56777a486
Sep 15 10:18:03.985301 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.985282    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z"
Sep 15 10:18:03.988718 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:03.988700    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33162a7a_bbbc_475d_b186_63e2b94656fe.slice/crio-901f5725879abc824c4c6e2d0c4847af444d0f6a5e803979e5212048136482cf WatchSource:0}: Error finding container 901f5725879abc824c4c6e2d0c4847af444d0f6a5e803979e5212048136482cf: Status 404 returned error can't find the container with id 901f5725879abc824c4c6e2d0c4847af444d0f6a5e803979e5212048136482cf
Sep 15 10:18:03.994058 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.993567    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 10:18:03.994058 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.993839    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zf2rw"
Sep 15 10:18:04.000005 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:03.999856    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q4zj5"
Sep 15 10:18:04.000097 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:04.000074    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92feea8_a50a_4580_bc28_281661e98802.slice/crio-810e5c1558199febfd0bf2470856527c6530cf39318e9ba69bd3f68e97122937 WatchSource:0}: Error finding container 810e5c1558199febfd0bf2470856527c6530cf39318e9ba69bd3f68e97122937: Status 404 returned error can't find the container with id 810e5c1558199febfd0bf2470856527c6530cf39318e9ba69bd3f68e97122937
Sep 15 10:18:04.008087 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:04.008021    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8246177_ac46_487b_9853_e979365daa1f.slice/crio-3405c8beb02e2f13b18ae415ed7395c0fe7b4853000a7bfabefec79da33938e9 WatchSource:0}: Error finding container 3405c8beb02e2f13b18ae415ed7395c0fe7b4853000a7bfabefec79da33938e9: Status 404 returned error can't find the container with id 3405c8beb02e2f13b18ae415ed7395c0fe7b4853000a7bfabefec79da33938e9
Sep 15 10:18:04.012859 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:04.012833    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e9ea1a_0387_4294_a760_df3980219768.slice/crio-5cdd1b9241af9feb7173cacc1adf7d0105b0ffca12edaea735e864d119dc4cf9 WatchSource:0}: Error finding container 5cdd1b9241af9feb7173cacc1adf7d0105b0ffca12edaea735e864d119dc4cf9: Status 404 returned error can't find the container with id 5cdd1b9241af9feb7173cacc1adf7d0105b0ffca12edaea735e864d119dc4cf9
Sep 15 10:18:04.356433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.356347    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kq6\" (UniqueName: \"kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6\") pod \"network-check-target-dszpz\" (UID: \"aab62d15-9d50-4842-9f39-0ed41176b31a\") " pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:04.356433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.356394    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:04.356433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.356428    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:04.356654 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:04.356526    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 10:18:04.356654 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:04.356540    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:04.356654 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:04.356556    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 10:18:04.356654 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:04.356569    2584 projected.go:196] Error preparing data for projected volume kube-api-access-x9kq6 for pod openshift-network-diagnostics/network-check-target-dszpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:04.356654 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:04.356532    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:04.356654 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:04.356608    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret podName:992c0c97-0af1-4197-82ca-e4c193a935f6 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:05.356587758 +0000 UTC m=+4.050743429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret") pod "global-pull-secret-syncer-txhqg" (UID: "992c0c97-0af1-4197-82ca-e4c193a935f6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:04.356654 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:04.356625    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6 podName:aab62d15-9d50-4842-9f39-0ed41176b31a nodeName:}" failed. No retries permitted until 2026-09-15 10:18:05.356616508 +0000 UTC m=+4.050772167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9kq6" (UniqueName: "kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6") pod "network-check-target-dszpz" (UID: "aab62d15-9d50-4842-9f39-0ed41176b31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:04.356654 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:04.356638    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs podName:6992d04d-f1f5-4521-a0da-27555d562676 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:05.356631421 +0000 UTC m=+4.050787079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs") pod "network-metrics-daemon-bftt2" (UID: "6992d04d-f1f5-4521-a0da-27555d562676") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:04.682722 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.682635    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 10:13:03 +0000 UTC" deadline="2028-03-29 09:21:24.628863316 +0000 UTC"
Sep 15 10:18:04.682722 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.682671    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13463h3m19.946196602s"
Sep 15 10:18:04.779689 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.779659    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:04.779869 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:04.779844    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:04.790560 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.790531    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cdxsk" event={"ID":"8b958d69-b275-4c2f-8f0a-31ba329cdc25","Type":"ContainerStarted","Data":"b809c12bd9988e5376d9bbe563a4f9fe4bfca669495f9b9df5a2fcc56777a486"}
Sep 15 10:18:04.792044 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.792018    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n8bbj" event={"ID":"a0582a9b-c2c1-437e-99a6-0a907acb9b35","Type":"ContainerStarted","Data":"ad15d2579e3edcaea0b0faf37459258b819803b3e77758d753b12c7f2bc424d6"}
Sep 15 10:18:04.793448 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.793414    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xhsgp" event={"ID":"f5bbf8a3-ce12-4124-9077-e892fef0316f","Type":"ContainerStarted","Data":"85b959c6409404019fff9d5cdea29e56c6da39ccdb4d9fa29f5cf62f857cd27b"}
Sep 15 10:18:04.794728 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.794701    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z" event={"ID":"f92feea8-a50a-4580-bc28-281661e98802","Type":"ContainerStarted","Data":"810e5c1558199febfd0bf2470856527c6530cf39318e9ba69bd3f68e97122937"}
Sep 15 10:18:04.795961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.795937    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgms7" event={"ID":"33162a7a-bbbc-475d-b186-63e2b94656fe","Type":"ContainerStarted","Data":"901f5725879abc824c4c6e2d0c4847af444d0f6a5e803979e5212048136482cf"}
Sep 15 10:18:04.797738 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.797710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8" event={"ID":"b996eb8a-182a-48ab-9aa6-6be28e14c8ac","Type":"ContainerStarted","Data":"8a3a05021f1cd1f32db426af294d9156d142ab3c215dbe1521167a21471be195"}
Sep 15 10:18:04.801409 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.801381    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7vfnf" event={"ID":"72c44564-2d7f-47fd-8567-d8793ed885bd","Type":"ContainerStarted","Data":"d1a952e1ad923ab84da5813f61b1e99e62b05b1a95d728cb67fa3981a8dcc063"}
Sep 15 10:18:04.802676 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.802648    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q4zj5" event={"ID":"b1e9ea1a-0387-4294-a760-df3980219768","Type":"ContainerStarted","Data":"5cdd1b9241af9feb7173cacc1adf7d0105b0ffca12edaea735e864d119dc4cf9"}
Sep 15 10:18:04.804049 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:04.804027    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zf2rw" event={"ID":"c8246177-ac46-487b-9853-e979365daa1f","Type":"ContainerStarted","Data":"3405c8beb02e2f13b18ae415ed7395c0fe7b4853000a7bfabefec79da33938e9"}
Sep 15 10:18:05.364407 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:05.364150    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kq6\" (UniqueName: \"kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6\") pod \"network-check-target-dszpz\" (UID: \"aab62d15-9d50-4842-9f39-0ed41176b31a\") " pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:05.364599 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:05.364435    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:05.364599 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:05.364453    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 10:18:05.364599 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:05.364482    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 10:18:05.364599 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:05.364498    2584 projected.go:196] Error preparing data for projected volume kube-api-access-x9kq6 for pod openshift-network-diagnostics/network-check-target-dszpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:05.364599 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:05.364559    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6 podName:aab62d15-9d50-4842-9f39-0ed41176b31a nodeName:}" failed. No retries permitted until 2026-09-15 10:18:07.364538547 +0000 UTC m=+6.058694220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9kq6" (UniqueName: "kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6") pod "network-check-target-dszpz" (UID: "aab62d15-9d50-4842-9f39-0ed41176b31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:05.364599 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:05.364570    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:05.364948 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:05.364623    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs podName:6992d04d-f1f5-4521-a0da-27555d562676 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:07.36460722 +0000 UTC m=+6.058762884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs") pod "network-metrics-daemon-bftt2" (UID: "6992d04d-f1f5-4521-a0da-27555d562676") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:05.364948 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:05.364683    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:05.364948 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:05.364717    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret podName:992c0c97-0af1-4197-82ca-e4c193a935f6 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:07.36470565 +0000 UTC m=+6.058861327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret") pod "global-pull-secret-syncer-txhqg" (UID: "992c0c97-0af1-4197-82ca-e4c193a935f6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:05.364948 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:05.364479    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:05.683822 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:05.683746    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 10:13:03 +0000 UTC" deadline="2028-06-01 09:53:57.574160124 +0000 UTC"
Sep 15 10:18:05.683822 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:05.683822    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14999h35m51.890343596s"
Sep 15 10:18:05.782497 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:05.781848    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:05.782497 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:05.781990    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:05.782497 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:05.782074    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:05.782497 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:05.782148    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:05.813635 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:05.813244    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-90.ec2.internal" event={"ID":"18620f5e88bb272825dcdf930e2ff0ad","Type":"ContainerStarted","Data":"32004291be404cea9f04faa4c0ffa2ab951ce697ae64eb0a6c43d5a03ebb2dc2"}
Sep 15 10:18:06.779781 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:06.779295    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:06.779781 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:06.779425    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:06.821112 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:06.820307    2584 generic.go:356] "Generic (PLEG): container finished" podID="d3c08f4f6e555d16caa02f9df8c01702" containerID="bf0908cb12ef7817a5e1f3cb4cd225627e9d3964850bca7e2dd4816180d03972" exitCode=0
Sep 15 10:18:06.821112 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:06.820889    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal" event={"ID":"d3c08f4f6e555d16caa02f9df8c01702","Type":"ContainerDied","Data":"bf0908cb12ef7817a5e1f3cb4cd225627e9d3964850bca7e2dd4816180d03972"}
Sep 15 10:18:06.835780 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:06.835674    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-90.ec2.internal" podStartSLOduration=4.835652993 podStartE2EDuration="4.835652993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:05.828093669 +0000 UTC m=+4.522249349" watchObservedRunningTime="2026-09-15 10:18:06.835652993 +0000 UTC m=+5.529808677"
Sep 15 10:18:07.384134 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:07.384099    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:07.384308 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:07.384182    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kq6\" (UniqueName: \"kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6\") pod \"network-check-target-dszpz\" (UID: \"aab62d15-9d50-4842-9f39-0ed41176b31a\") " pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:07.384308 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:07.384212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:07.384430 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:07.384332    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:07.384430 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:07.384391    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret podName:992c0c97-0af1-4197-82ca-e4c193a935f6 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:11.384374572 +0000 UTC m=+10.078530233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret") pod "global-pull-secret-syncer-txhqg" (UID: "992c0c97-0af1-4197-82ca-e4c193a935f6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:07.384791 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:07.384772    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:07.384879 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:07.384816    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 10:18:07.384879 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:07.384841    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs podName:6992d04d-f1f5-4521-a0da-27555d562676 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:11.384826361 +0000 UTC m=+10.078982024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs") pod "network-metrics-daemon-bftt2" (UID: "6992d04d-f1f5-4521-a0da-27555d562676") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:07.384879 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:07.384846    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 10:18:07.384879 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:07.384860    2584 projected.go:196] Error preparing data for projected volume kube-api-access-x9kq6 for pod openshift-network-diagnostics/network-check-target-dszpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:07.385082 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:07.384912    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6 podName:aab62d15-9d50-4842-9f39-0ed41176b31a nodeName:}" failed. No retries permitted until 2026-09-15 10:18:11.384895998 +0000 UTC m=+10.079051670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9kq6" (UniqueName: "kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6") pod "network-check-target-dszpz" (UID: "aab62d15-9d50-4842-9f39-0ed41176b31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:07.789722 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:07.787904    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:07.789722 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:07.788084    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:07.789722 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:07.788925    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:07.789722 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:07.789086    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:08.779522 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:08.779460    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:08.779743 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:08.779592    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:09.778653 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:09.778619    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:09.779011 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:09.778766    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:09.779011 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:09.778820    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:09.779011 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:09.778914    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:10.779109 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:10.779031    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:10.779558 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:10.779165    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:11.421749 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:11.421705    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:11.421946 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:11.421812    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kq6\" (UniqueName: \"kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6\") pod \"network-check-target-dszpz\" (UID: \"aab62d15-9d50-4842-9f39-0ed41176b31a\") " pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:11.421946 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:11.421886    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:11.421946 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:11.421923    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 10:18:11.421946 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:11.421940    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 10:18:11.422132 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:11.421953    2584 projected.go:196] Error preparing data for projected volume kube-api-access-x9kq6 for pod openshift-network-diagnostics/network-check-target-dszpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:11.422132 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:11.421961    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs podName:6992d04d-f1f5-4521-a0da-27555d562676 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:19.421939632 +0000 UTC m=+18.116095303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs") pod "network-metrics-daemon-bftt2" (UID: "6992d04d-f1f5-4521-a0da-27555d562676") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:11.422132 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:11.421987    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:11.422132 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:11.421921    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:11.422132 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:11.422002    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6 podName:aab62d15-9d50-4842-9f39-0ed41176b31a nodeName:}" failed. No retries permitted until 2026-09-15 10:18:19.421986157 +0000 UTC m=+18.116141831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9kq6" (UniqueName: "kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6") pod "network-check-target-dszpz" (UID: "aab62d15-9d50-4842-9f39-0ed41176b31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:11.422132 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:11.422022    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret podName:992c0c97-0af1-4197-82ca-e4c193a935f6 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:19.422012184 +0000 UTC m=+18.116167856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret") pod "global-pull-secret-syncer-txhqg" (UID: "992c0c97-0af1-4197-82ca-e4c193a935f6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:11.784387 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:11.781687    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:11.784387 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:11.781859    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:11.784387 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:11.782335    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:11.784387 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:11.782482    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:12.778937 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:12.778901    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:12.779149 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:12.779032    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:13.779339 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:13.779301    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:13.779991 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:13.779439    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:13.779991 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:13.779492    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:13.779991 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:13.779621    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:14.779044 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:14.779005    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:14.779199 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:14.779148    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:15.782375 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:15.782347    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:15.782833 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:15.782480    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:15.782833 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:15.782537    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:15.782833 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:15.782634    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:16.779508 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:16.779475    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:16.779692 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:16.779579    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:17.778789 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:17.778751    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:17.779265 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:17.778915    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:17.779265 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:17.778976    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:17.779265 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:17.779084    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:18.778947 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:18.778911    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:18.779393 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:18.779036    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:19.480275 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:19.480231    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kq6\" (UniqueName: \"kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6\") pod \"network-check-target-dszpz\" (UID: \"aab62d15-9d50-4842-9f39-0ed41176b31a\") " pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:19.480456 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:19.480286    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:19.480456 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:19.480407    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:19.480456 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:19.480434    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 10:18:19.480630 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:19.480461    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 10:18:19.480630 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:19.480475    2584 projected.go:196] Error preparing data for projected volume kube-api-access-x9kq6 for pod openshift-network-diagnostics/network-check-target-dszpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:19.480630 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:19.480471    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:19.480630 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:19.480527    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6 podName:aab62d15-9d50-4842-9f39-0ed41176b31a nodeName:}" failed. No retries permitted until 2026-09-15 10:18:35.480510357 +0000 UTC m=+34.174666035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9kq6" (UniqueName: "kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6") pod "network-check-target-dszpz" (UID: "aab62d15-9d50-4842-9f39-0ed41176b31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:19.480630 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:19.480545    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret podName:992c0c97-0af1-4197-82ca-e4c193a935f6 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:35.480535498 +0000 UTC m=+34.174691159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret") pod "global-pull-secret-syncer-txhqg" (UID: "992c0c97-0af1-4197-82ca-e4c193a935f6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:19.480630 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:19.480579    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:19.480630 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:19.480625    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs podName:6992d04d-f1f5-4521-a0da-27555d562676 nodeName:}" failed. No retries permitted until 2026-09-15 10:18:35.480614625 +0000 UTC m=+34.174770301 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs") pod "network-metrics-daemon-bftt2" (UID: "6992d04d-f1f5-4521-a0da-27555d562676") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:19.779156 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:19.779062    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:19.779644 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:19.779080    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:19.779644 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:19.779215    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:19.779644 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:19.779297    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:20.779177 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:20.779143    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:20.779623 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:20.779248    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:21.782012 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:21.781985    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:21.782358 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:21.781985    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:21.782358 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:21.782087    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:21.782358 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:21.782139    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:22.779555 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.779216    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:22.779724 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:22.779688    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:22.855795 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.855299    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal" event={"ID":"d3c08f4f6e555d16caa02f9df8c01702","Type":"ContainerStarted","Data":"8f7a525659c9e0463eab770d864c49c87d602d0f8f7dddd84197985f7aac395f"}
Sep 15 10:18:22.857374 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.857345    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z" event={"ID":"f92feea8-a50a-4580-bc28-281661e98802","Type":"ContainerStarted","Data":"0d7b2a0829a3b359812de66f03bbc21dc5a4f6db5714d2175ad6dd7d703b96eb"}
Sep 15 10:18:22.859673 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.859628    2584 generic.go:356] "Generic (PLEG): container finished" podID="33162a7a-bbbc-475d-b186-63e2b94656fe" containerID="36920cdadf65cebbb521d8a4147d1ffac1f5a763a329ac7623c81635a4c4cd37" exitCode=0
Sep 15 10:18:22.859787 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.859705    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgms7" event={"ID":"33162a7a-bbbc-475d-b186-63e2b94656fe","Type":"ContainerDied","Data":"36920cdadf65cebbb521d8a4147d1ffac1f5a763a329ac7623c81635a4c4cd37"}
Sep 15 10:18:22.863434 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.863412    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8" event={"ID":"b996eb8a-182a-48ab-9aa6-6be28e14c8ac","Type":"ContainerStarted","Data":"dec053ed58f3e00a70654603f7a61fe2877e741dfc4f62885eeda6a8fb713c9b"}
Sep 15 10:18:22.863538 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.863484    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8" event={"ID":"b996eb8a-182a-48ab-9aa6-6be28e14c8ac","Type":"ContainerStarted","Data":"04f9a6d66ec1783fbb2ab6c7e9c379d1db299c0743915e68d565e8c05f0215d3"}
Sep 15 10:18:22.863538 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.863536    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8" event={"ID":"b996eb8a-182a-48ab-9aa6-6be28e14c8ac","Type":"ContainerStarted","Data":"cd6a9befbdd20728a969f469972f9e7add4251ff30a928e003314361cbaf571d"}
Sep 15 10:18:22.863787 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.863627    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8" event={"ID":"b996eb8a-182a-48ab-9aa6-6be28e14c8ac","Type":"ContainerStarted","Data":"0b78b3cf9d42f38fc4d25b1814b9b0ada2474e995ed2b33fcf85c5d64b73881d"}
Sep 15 10:18:22.863787 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.863695    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8" event={"ID":"b996eb8a-182a-48ab-9aa6-6be28e14c8ac","Type":"ContainerStarted","Data":"8b6b300de240b619323f00c602c96c8166f2a3eafff5ce197a9154ad9622786e"}
Sep 15 10:18:22.863787 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.863742    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8" event={"ID":"b996eb8a-182a-48ab-9aa6-6be28e14c8ac","Type":"ContainerStarted","Data":"be5c81bbc4bcba08e540e688ad2bbf561f59d44bea6636ce96f41e4cb44c356d"}
Sep 15 10:18:22.865472 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.865430    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7vfnf" event={"ID":"72c44564-2d7f-47fd-8567-d8793ed885bd","Type":"ContainerStarted","Data":"46fe7a8b8d968a6ced4a495d608da3319aa9c24789b0ca10e65c596cce412755"}
Sep 15 10:18:22.867216 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.867193    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q4zj5" event={"ID":"b1e9ea1a-0387-4294-a760-df3980219768","Type":"ContainerStarted","Data":"1744d8287fd6c47bdc6a5151acd44238b46a1c1e08f5b1fc03498fcdd18c9ced"}
Sep 15 10:18:22.868866 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.868830    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zf2rw" event={"ID":"c8246177-ac46-487b-9853-e979365daa1f","Type":"ContainerStarted","Data":"e1f49d8e5d93c376dd00dbb1dc24778f24965cc9db8f826749ea6f2a66de6a9d"}
Sep 15 10:18:22.869428 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.869381    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-90.ec2.internal" podStartSLOduration=20.869365098 podStartE2EDuration="20.869365098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:22.869015915 +0000 UTC m=+21.563171596" watchObservedRunningTime="2026-09-15 10:18:22.869365098 +0000 UTC m=+21.563520778"
Sep 15 10:18:22.870759 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.870735    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cdxsk" event={"ID":"8b958d69-b275-4c2f-8f0a-31ba329cdc25","Type":"ContainerStarted","Data":"180ab0099a9a10ae62931c69e83f222749f4e9b77071d3750d05a08e331d25b3"}
Sep 15 10:18:22.872781 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.872760    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n8bbj" event={"ID":"a0582a9b-c2c1-437e-99a6-0a907acb9b35","Type":"ContainerStarted","Data":"dabce056f600d05379064425b791f4b282895f37846f3c9463f32b51e0edd6f3"}
Sep 15 10:18:22.881538 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.881504    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7vfnf" podStartSLOduration=3.978509615 podStartE2EDuration="21.881493404s" totalImagesPullingTime="17.902983789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:22.88106907 +0000 UTC m=+21.575224751" watchObservedRunningTime="2026-09-15 10:18:22.881493404 +0000 UTC m=+21.575649083"
Sep 15 10:18:22.927192 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.927139    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q4zj5" podStartSLOduration=4.086325117 podStartE2EDuration="21.927122159s" totalImagesPullingTime="17.840797042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:22.912007061 +0000 UTC m=+21.606162742" watchObservedRunningTime="2026-09-15 10:18:22.927122159 +0000 UTC m=+21.621277840"
Sep 15 10:18:22.927328 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.927242    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zf2rw" podStartSLOduration=3.995954128 podStartE2EDuration="21.927233444s" totalImagesPullingTime="17.931279316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:22.92712111 +0000 UTC m=+21.621276790" watchObservedRunningTime="2026-09-15 10:18:22.927233444 +0000 UTC m=+21.621389123"
Sep 15 10:18:22.940161 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.940098    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n8bbj" podStartSLOduration=8.025129093 podStartE2EDuration="21.940083165s" totalImagesPullingTime="13.914954072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:22.939378407 +0000 UTC m=+21.633534085" watchObservedRunningTime="2026-09-15 10:18:22.940083165 +0000 UTC m=+21.634238893"
Sep 15 10:18:22.957173 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:22.957126    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cdxsk" podStartSLOduration=4.086829332 podStartE2EDuration="21.957113091s" totalImagesPullingTime="17.870283759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:22.956519107 +0000 UTC m=+21.650674778" watchObservedRunningTime="2026-09-15 10:18:22.957113091 +0000 UTC m=+21.651268770"
Sep 15 10:18:23.175857 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:23.175830    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 10:18:23.714677 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:23.714572    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T10:18:23.175851559Z","UUID":"7abba18a-b7ac-4a1b-9914-c8abf0b2a787","Handler":null,"Name":"","Endpoint":""}
Sep 15 10:18:23.716317 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:23.716291    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 10:18:23.716317 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:23.716324    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 10:18:23.779079 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:23.779049    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:23.779241 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:23.779164    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:23.779483 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:23.779049    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:23.779483 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:23.779444    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:23.876908 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:23.876842    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z" event={"ID":"f92feea8-a50a-4580-bc28-281661e98802","Type":"ContainerStarted","Data":"15439d9ee7107cc5cc0677a21d990bf1826cbac7c582613c16f3d68d04ebbc4b"}
Sep 15 10:18:23.878433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:23.878406    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xhsgp" event={"ID":"f5bbf8a3-ce12-4124-9077-e892fef0316f","Type":"ContainerStarted","Data":"aa6c18f73f75f169529e4d7587da9f49631eafcfca5fd214aa570ed486cd7653"}
Sep 15 10:18:23.896934 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:23.896895    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xhsgp" podStartSLOduration=5.000264914 podStartE2EDuration="22.896880005s" totalImagesPullingTime="17.896615091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:23.89627524 +0000 UTC m=+22.590430921" watchObservedRunningTime="2026-09-15 10:18:23.896880005 +0000 UTC m=+22.591035684"
Sep 15 10:18:24.779198 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:24.779117    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:24.779359 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:24.779246    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:24.883785 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:24.883753    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z" event={"ID":"f92feea8-a50a-4580-bc28-281661e98802","Type":"ContainerStarted","Data":"e494723109770babf11280af23e39b22155c9115f9364f93a80ef9a473605458"}
Sep 15 10:18:24.891955 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:24.891922    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8" event={"ID":"b996eb8a-182a-48ab-9aa6-6be28e14c8ac","Type":"ContainerStarted","Data":"36577474c6887d67094b705594fb965788f397d48de4afd34d04323889af9a76"}
Sep 15 10:18:24.902582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:24.902527    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2xq8z" podStartSLOduration=3.804519597 podStartE2EDuration="23.902505617s" totalImagesPullingTime="20.09798602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:24.902409434 +0000 UTC m=+23.596565125" watchObservedRunningTime="2026-09-15 10:18:24.902505617 +0000 UTC m=+23.596661297"
Sep 15 10:18:25.781673 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:25.781646    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:25.781943 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:25.781721    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:25.781943 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:25.781742    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:25.781943 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:25.781850    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:26.779587 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:26.779340    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:26.780018 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:26.779696    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:27.585227 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.585191    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:27.586010 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.585987    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:27.779364 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.779284    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:27.779509 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:27.779413    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:27.779509 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.779464    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:27.779585 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:27.779551    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:27.900817 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.900627    2584 generic.go:356] "Generic (PLEG): container finished" podID="33162a7a-bbbc-475d-b186-63e2b94656fe" containerID="8e961e0e6af9573d037044fb729a6d52330258f0e054014744606eb0d89782b4" exitCode=0
Sep 15 10:18:27.900817 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.900708    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgms7" event={"ID":"33162a7a-bbbc-475d-b186-63e2b94656fe","Type":"ContainerDied","Data":"8e961e0e6af9573d037044fb729a6d52330258f0e054014744606eb0d89782b4"}
Sep 15 10:18:27.904649 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.904628    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8" event={"ID":"b996eb8a-182a-48ab-9aa6-6be28e14c8ac","Type":"ContainerStarted","Data":"3d585e99c210af051d2fac295040682ac01da7ad23fe25141a8650d0b44cfe89"}
Sep 15 10:18:27.904985 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.904948    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:27.905085 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.905018    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:27.905085 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.905071    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:27.905232 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.905101    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:27.905691 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.905675    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n8bbj"
Sep 15 10:18:27.920163 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.920139    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:27.920269 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.920238    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:18:27.950142 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:27.950098    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8" podStartSLOduration=8.512001556 podStartE2EDuration="26.950085621s" totalImagesPullingTime="18.438084065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:27.949611516 +0000 UTC m=+26.643767257" watchObservedRunningTime="2026-09-15 10:18:27.950085621 +0000 UTC m=+26.644241300"
Sep 15 10:18:28.778652 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:28.778616    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:28.778824 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:28.778719    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:29.307259 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:29.307228    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dszpz"]
Sep 15 10:18:29.308007 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:29.307344    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:29.308007 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:29.307440    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:29.310105 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:29.310083    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-txhqg"]
Sep 15 10:18:29.310221 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:29.310187    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:29.310306 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:29.310284    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:29.310791 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:29.310766    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bftt2"]
Sep 15 10:18:29.310917 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:29.310906    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:29.311039 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:29.311017    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:29.911243 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:29.911210    2584 generic.go:356] "Generic (PLEG): container finished" podID="33162a7a-bbbc-475d-b186-63e2b94656fe" containerID="3d3bcc83ade4e2e53412cdc82dff3f7d69743895ac4dd5dc66af558df8eae373" exitCode=0
Sep 15 10:18:29.911395 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:29.911297    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgms7" event={"ID":"33162a7a-bbbc-475d-b186-63e2b94656fe","Type":"ContainerDied","Data":"3d3bcc83ade4e2e53412cdc82dff3f7d69743895ac4dd5dc66af558df8eae373"}
Sep 15 10:18:30.779646 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:30.779534    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:30.779646 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:30.779594    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:30.779646 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:30.779563    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:30.780185 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:30.779693    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:30.780185 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:30.779750    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:30.780185 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:30.779840    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:30.916388 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:30.916355    2584 generic.go:356] "Generic (PLEG): container finished" podID="33162a7a-bbbc-475d-b186-63e2b94656fe" containerID="077a6e059da45fca85beb24ad3c4f35d798e8775299ed66c09807b39223af97a" exitCode=0
Sep 15 10:18:30.916535 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:30.916406    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgms7" event={"ID":"33162a7a-bbbc-475d-b186-63e2b94656fe","Type":"ContainerDied","Data":"077a6e059da45fca85beb24ad3c4f35d798e8775299ed66c09807b39223af97a"}
Sep 15 10:18:32.778973 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:32.778707    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:32.779460 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:32.778711    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:32.779460 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:32.778720    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:32.779460 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:32.779086    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:32.779460 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:32.779240    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:32.779460 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:32.779297    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:34.779686 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:34.779645    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:34.780140 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:34.779663    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:34.780140 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:34.779780    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:34.780140 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:34.779664    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:34.780140 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:34.779869    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:34.780140 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:34.779968    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:35.496123 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:35.496084    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kq6\" (UniqueName: \"kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6\") pod \"network-check-target-dszpz\" (UID: \"aab62d15-9d50-4842-9f39-0ed41176b31a\") " pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:35.496345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:35.496132    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:35.496345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:35.496165    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:35.496345 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:35.496268    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:35.496345 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:35.496331    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs podName:6992d04d-f1f5-4521-a0da-27555d562676 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:07.49631535 +0000 UTC m=+66.190471021 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs") pod "network-metrics-daemon-bftt2" (UID: "6992d04d-f1f5-4521-a0da-27555d562676") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 10:18:35.496541 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:35.496268    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:35.496541 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:35.496411    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret podName:992c0c97-0af1-4197-82ca-e4c193a935f6 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:07.496398676 +0000 UTC m=+66.190554348 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret") pod "global-pull-secret-syncer-txhqg" (UID: "992c0c97-0af1-4197-82ca-e4c193a935f6") : object "kube-system"/"original-pull-secret" not registered
Sep 15 10:18:35.496541 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:35.496267    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 10:18:35.496541 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:35.496438    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 10:18:35.496541 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:35.496452    2584 projected.go:196] Error preparing data for projected volume kube-api-access-x9kq6 for pod openshift-network-diagnostics/network-check-target-dszpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:35.496541 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:35.496490    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6 podName:aab62d15-9d50-4842-9f39-0ed41176b31a nodeName:}" failed. No retries permitted until 2026-09-15 10:19:07.496480219 +0000 UTC m=+66.190635893 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9kq6" (UniqueName: "kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6") pod "network-check-target-dszpz" (UID: "aab62d15-9d50-4842-9f39-0ed41176b31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 10:18:36.779049 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:36.778963    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:36.779049 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:36.778963    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:36.779454 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:36.779088    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:36.779454 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:36.779100    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:36.779454 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:36.779159    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:36.779454 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:36.779232    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:36.932092 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:36.932059    2584 generic.go:356] "Generic (PLEG): container finished" podID="33162a7a-bbbc-475d-b186-63e2b94656fe" containerID="ea0faabf9f7e4c1f95827ecddf9f071b1ca982a2210ce78e213f7bde1cdbbbfe" exitCode=0
Sep 15 10:18:36.932243 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:36.932106    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgms7" event={"ID":"33162a7a-bbbc-475d-b186-63e2b94656fe","Type":"ContainerDied","Data":"ea0faabf9f7e4c1f95827ecddf9f071b1ca982a2210ce78e213f7bde1cdbbbfe"}
Sep 15 10:18:37.937983 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:37.937947    2584 generic.go:356] "Generic (PLEG): container finished" podID="33162a7a-bbbc-475d-b186-63e2b94656fe" containerID="1b192a75543623902c504dadec61da710f9dcfb0f5471d45cd82b50e86b6da93" exitCode=0
Sep 15 10:18:37.938362 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:37.938007    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgms7" event={"ID":"33162a7a-bbbc-475d-b186-63e2b94656fe","Type":"ContainerDied","Data":"1b192a75543623902c504dadec61da710f9dcfb0f5471d45cd82b50e86b6da93"}
Sep 15 10:18:38.779680 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:38.779485    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:38.779862 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:38.779487    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:38.779862 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:38.779766    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:38.779942 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:38.779855    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:38.779942 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:38.779532    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:38.779942 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:38.779936    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:38.944068 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:38.944029    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgms7" event={"ID":"33162a7a-bbbc-475d-b186-63e2b94656fe","Type":"ContainerStarted","Data":"c349c30159f836640d706eb0105d633429da4d9ddf427edd989bcbeaf5725ed5"}
Sep 15 10:18:38.966555 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:38.966503    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bgms7" podStartSLOduration=8.131083973 podStartE2EDuration="37.966484176s" totalImagesPullingTime="29.835400203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:38.965907333 +0000 UTC m=+37.660063012" watchObservedRunningTime="2026-09-15 10:18:38.966484176 +0000 UTC m=+37.660639853"
Sep 15 10:18:40.778996 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:40.778949    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:40.778996 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:40.779002    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:40.779417 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:40.779006    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:40.779417 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:40.779075    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:40.779417 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:40.779157    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:40.779417 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:40.779238    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:42.779032 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:42.779002    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:42.779403 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:42.779100    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:42.779403 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:42.779122    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:42.779403 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:42.779112    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:42.779403 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:42.779200    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:42.779403 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:42.779287    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:44.778664 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:44.778624    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:44.779183 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:44.778624    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:44.779183 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:44.778626    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:44.779183 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:44.778720    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:44.779183 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:44.778902    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:44.779183 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:44.778959    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:46.779436 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:46.779400    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:46.779436 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:46.779438    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:46.779892 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:46.779520    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:46.779892 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:46.779562    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:46.779892 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:46.779606    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:46.779892 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:46.779696    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:48.779534 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:48.779501    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:48.779939 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:48.779602    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:48.779939 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:48.779612    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:48.779939 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:48.779623    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:48.779939 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:48.779691    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:48.779939 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:48.779771    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:50.779305 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:50.779258    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:50.779305 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:50.779281    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:50.779855 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:50.779371    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:50.779855 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:50.779426    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:50.779855 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:50.779539    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:50.779855 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:50.779635    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:52.779639 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:52.779604    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:52.780192 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:52.779601    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:52.780192 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:52.779708    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dszpz" podUID="aab62d15-9d50-4842-9f39-0ed41176b31a"
Sep 15 10:18:52.780192 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:52.779604    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:52.780192 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:52.779817    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bftt2" podUID="6992d04d-f1f5-4521-a0da-27555d562676"
Sep 15 10:18:52.780192 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:18:52.779921    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txhqg" podUID="992c0c97-0af1-4197-82ca-e4c193a935f6"
Sep 15 10:18:54.043283 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.043255    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-90.ec2.internal" event="NodeReady"
Sep 15 10:18:54.043671 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.043381    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 10:18:54.078501 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.078466    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79dd5bfd77-hrtl6"]
Sep 15 10:18:54.080735 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.080717    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.082764 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.082734    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zhqkr\""
Sep 15 10:18:54.082948 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.082905    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 10:18:54.083015 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.082981    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 10:18:54.083015 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.082994    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 10:18:54.087844 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.087794    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 10:18:54.093505 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.093476    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79dd5bfd77-hrtl6"]
Sep 15 10:18:54.097874 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.097849    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gcxvj"]
Sep 15 10:18:54.100077 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.100056    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hwtxj"]
Sep 15 10:18:54.100189 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.100170    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.102069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.102049    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hwtxj"
Sep 15 10:18:54.103541 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.103522    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 10:18:54.103936 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.103920    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 10:18:54.104158 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.104133    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 10:18:54.104260 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.104200    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 10:18:54.104372 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.104327    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 10:18:54.104372 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.104340    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4q5lf\""
Sep 15 10:18:54.104568 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.104550    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h7d4n\""
Sep 15 10:18:54.107955 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.107896    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hwtxj"]
Sep 15 10:18:54.110248 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.110229    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gcxvj"]
Sep 15 10:18:54.124776 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.124746    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1c396c5-e224-478f-a09d-6857eee1ae0e-registry-tls\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.124776 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.124776    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7vb4\" (UniqueName: \"kubernetes.io/projected/42c7db80-61f8-46e5-8458-3db6c66a56c3-kube-api-access-g7vb4\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.124978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.124792    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/146f0ea1-aecd-451f-b1f1-8bc242b439d8-cert\") pod \"ingress-canary-hwtxj\" (UID: \"146f0ea1-aecd-451f-b1f1-8bc242b439d8\") " pod="openshift-ingress-canary/ingress-canary-hwtxj"
Sep 15 10:18:54.124978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.124836    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1c396c5-e224-478f-a09d-6857eee1ae0e-registry-certificates\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.124978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.124867    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42c7db80-61f8-46e5-8458-3db6c66a56c3-config-volume\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.124978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.124905    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1c396c5-e224-478f-a09d-6857eee1ae0e-installation-pull-secrets\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.124978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.124928    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42c7db80-61f8-46e5-8458-3db6c66a56c3-metrics-tls\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.124978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.124947    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85w9q\" (UniqueName: \"kubernetes.io/projected/146f0ea1-aecd-451f-b1f1-8bc242b439d8-kube-api-access-85w9q\") pod \"ingress-canary-hwtxj\" (UID: \"146f0ea1-aecd-451f-b1f1-8bc242b439d8\") " pod="openshift-ingress-canary/ingress-canary-hwtxj"
Sep 15 10:18:54.124978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.124966    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1c396c5-e224-478f-a09d-6857eee1ae0e-ca-trust-extracted\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.125187 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.125027    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c7db80-61f8-46e5-8458-3db6c66a56c3-tmp-dir\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.125187 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.125063    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1c396c5-e224-478f-a09d-6857eee1ae0e-bound-sa-token\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.125187 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.125076    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p2kx\" (UniqueName: \"kubernetes.io/projected/f1c396c5-e224-478f-a09d-6857eee1ae0e-kube-api-access-9p2kx\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.125187 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.125101    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1c396c5-e224-478f-a09d-6857eee1ae0e-trusted-ca\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.125187 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.125123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1c396c5-e224-478f-a09d-6857eee1ae0e-image-registry-private-configuration\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.225381 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.225337    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1c396c5-e224-478f-a09d-6857eee1ae0e-installation-pull-secrets\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.225381 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.225384    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42c7db80-61f8-46e5-8458-3db6c66a56c3-metrics-tls\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.225615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.225416    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85w9q\" (UniqueName: \"kubernetes.io/projected/146f0ea1-aecd-451f-b1f1-8bc242b439d8-kube-api-access-85w9q\") pod \"ingress-canary-hwtxj\" (UID: \"146f0ea1-aecd-451f-b1f1-8bc242b439d8\") " pod="openshift-ingress-canary/ingress-canary-hwtxj"
Sep 15 10:18:54.225615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.225443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1c396c5-e224-478f-a09d-6857eee1ae0e-ca-trust-extracted\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.225615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.225474    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c7db80-61f8-46e5-8458-3db6c66a56c3-tmp-dir\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.225615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.225525    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1c396c5-e224-478f-a09d-6857eee1ae0e-bound-sa-token\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.225615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.225549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p2kx\" (UniqueName: \"kubernetes.io/projected/f1c396c5-e224-478f-a09d-6857eee1ae0e-kube-api-access-9p2kx\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.225615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.225587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1c396c5-e224-478f-a09d-6857eee1ae0e-trusted-ca\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.225982 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.225955    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c7db80-61f8-46e5-8458-3db6c66a56c3-tmp-dir\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.225982 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.225975    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1c396c5-e224-478f-a09d-6857eee1ae0e-image-registry-private-configuration\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.226109 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.225983    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1c396c5-e224-478f-a09d-6857eee1ae0e-ca-trust-extracted\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.226109 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.226019    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1c396c5-e224-478f-a09d-6857eee1ae0e-registry-tls\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.226109 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.226047    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7vb4\" (UniqueName: \"kubernetes.io/projected/42c7db80-61f8-46e5-8458-3db6c66a56c3-kube-api-access-g7vb4\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.226109 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.226070    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/146f0ea1-aecd-451f-b1f1-8bc242b439d8-cert\") pod \"ingress-canary-hwtxj\" (UID: \"146f0ea1-aecd-451f-b1f1-8bc242b439d8\") " pod="openshift-ingress-canary/ingress-canary-hwtxj"
Sep 15 10:18:54.226305 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.226113    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1c396c5-e224-478f-a09d-6857eee1ae0e-registry-certificates\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.226305 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.226136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42c7db80-61f8-46e5-8458-3db6c66a56c3-config-volume\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.226605 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.226583    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1c396c5-e224-478f-a09d-6857eee1ae0e-trusted-ca\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.226813 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.226651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42c7db80-61f8-46e5-8458-3db6c66a56c3-config-volume\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.227228 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.227200    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1c396c5-e224-478f-a09d-6857eee1ae0e-registry-certificates\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.229892 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.229864    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1c396c5-e224-478f-a09d-6857eee1ae0e-image-registry-private-configuration\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.229975 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.229889    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42c7db80-61f8-46e5-8458-3db6c66a56c3-metrics-tls\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.229975 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.229891    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1c396c5-e224-478f-a09d-6857eee1ae0e-installation-pull-secrets\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.229975 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.229942    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1c396c5-e224-478f-a09d-6857eee1ae0e-registry-tls\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.229975 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.229951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/146f0ea1-aecd-451f-b1f1-8bc242b439d8-cert\") pod \"ingress-canary-hwtxj\" (UID: \"146f0ea1-aecd-451f-b1f1-8bc242b439d8\") " pod="openshift-ingress-canary/ingress-canary-hwtxj"
Sep 15 10:18:54.241727 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.241700    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85w9q\" (UniqueName: \"kubernetes.io/projected/146f0ea1-aecd-451f-b1f1-8bc242b439d8-kube-api-access-85w9q\") pod \"ingress-canary-hwtxj\" (UID: \"146f0ea1-aecd-451f-b1f1-8bc242b439d8\") " pod="openshift-ingress-canary/ingress-canary-hwtxj"
Sep 15 10:18:54.241872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.241748    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p2kx\" (UniqueName: \"kubernetes.io/projected/f1c396c5-e224-478f-a09d-6857eee1ae0e-kube-api-access-9p2kx\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.242159 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.242141    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1c396c5-e224-478f-a09d-6857eee1ae0e-bound-sa-token\") pod \"image-registry-79dd5bfd77-hrtl6\" (UID: \"f1c396c5-e224-478f-a09d-6857eee1ae0e\") " pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.242234 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.242186    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7vb4\" (UniqueName: \"kubernetes.io/projected/42c7db80-61f8-46e5-8458-3db6c66a56c3-kube-api-access-g7vb4\") pod \"dns-default-gcxvj\" (UID: \"42c7db80-61f8-46e5-8458-3db6c66a56c3\") " pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.390219 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.390116    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.410540 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.410508    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:54.416098 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.416068    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hwtxj"
Sep 15 10:18:54.690463 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.690388    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79dd5bfd77-hrtl6"]
Sep 15 10:18:54.693786 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:54.693760    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c396c5_e224_478f_a09d_6857eee1ae0e.slice/crio-fc5d4db192aea6fc74f2e555041c7d91e900e4352df75c026dacd72a8bd62585 WatchSource:0}: Error finding container fc5d4db192aea6fc74f2e555041c7d91e900e4352df75c026dacd72a8bd62585: Status 404 returned error can't find the container with id fc5d4db192aea6fc74f2e555041c7d91e900e4352df75c026dacd72a8bd62585
Sep 15 10:18:54.703479 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.703457    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gcxvj"]
Sep 15 10:18:54.706408 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.706293    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hwtxj"]
Sep 15 10:18:54.707018 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:54.706993    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c7db80_61f8_46e5_8458_3db6c66a56c3.slice/crio-08d5825089f5a95893930cfc2e001ccf1b1bcc5bf92a41ae5dd57a702b94ceaf WatchSource:0}: Error finding container 08d5825089f5a95893930cfc2e001ccf1b1bcc5bf92a41ae5dd57a702b94ceaf: Status 404 returned error can't find the container with id 08d5825089f5a95893930cfc2e001ccf1b1bcc5bf92a41ae5dd57a702b94ceaf
Sep 15 10:18:54.709892 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:18:54.709870    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146f0ea1_aecd_451f_b1f1_8bc242b439d8.slice/crio-a9afc6e40c6e15fd464764331b1e3e52429bf74f4a829e50ebd322d47347bdbf WatchSource:0}: Error finding container a9afc6e40c6e15fd464764331b1e3e52429bf74f4a829e50ebd322d47347bdbf: Status 404 returned error can't find the container with id a9afc6e40c6e15fd464764331b1e3e52429bf74f4a829e50ebd322d47347bdbf
Sep 15 10:18:54.779199 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.779172    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:18:54.779340 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.779210    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:18:54.779340 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.779287    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:18:54.781819 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.781786    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 10:18:54.781923 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.781875    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 10:18:54.781923 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.781911    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 10:18:54.782516 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.782487    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lrdhd\""
Sep 15 10:18:54.782606 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.782518    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-88p68\""
Sep 15 10:18:54.782606 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.782582    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 10:18:54.983695 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.983606    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gcxvj" event={"ID":"42c7db80-61f8-46e5-8458-3db6c66a56c3","Type":"ContainerStarted","Data":"08d5825089f5a95893930cfc2e001ccf1b1bcc5bf92a41ae5dd57a702b94ceaf"}
Sep 15 10:18:54.985132 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.985102    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6" event={"ID":"f1c396c5-e224-478f-a09d-6857eee1ae0e","Type":"ContainerStarted","Data":"0784c5f12828a358860f822a21112305ce75032a6d1013ff9d724e037b140a82"}
Sep 15 10:18:54.985271 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.985158    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6" event={"ID":"f1c396c5-e224-478f-a09d-6857eee1ae0e","Type":"ContainerStarted","Data":"fc5d4db192aea6fc74f2e555041c7d91e900e4352df75c026dacd72a8bd62585"}
Sep 15 10:18:54.985271 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.985184    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:18:54.986329 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:54.986307    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hwtxj" event={"ID":"146f0ea1-aecd-451f-b1f1-8bc242b439d8","Type":"ContainerStarted","Data":"a9afc6e40c6e15fd464764331b1e3e52429bf74f4a829e50ebd322d47347bdbf"}
Sep 15 10:18:55.003905 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:55.003855    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6" podStartSLOduration=2.003841 podStartE2EDuration="2.003841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:55.003725964 +0000 UTC m=+53.697881645" watchObservedRunningTime="2026-09-15 10:18:55.003841 +0000 UTC m=+53.697996679"
Sep 15 10:18:56.995704 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:56.995666    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gcxvj" event={"ID":"42c7db80-61f8-46e5-8458-3db6c66a56c3","Type":"ContainerStarted","Data":"1a52a9adbda2ea0665ddc601e4a0b8b79f1387f20323bd86ecc70bde3a7039b0"}
Sep 15 10:18:56.996177 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:56.995730    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gcxvj" event={"ID":"42c7db80-61f8-46e5-8458-3db6c66a56c3","Type":"ContainerStarted","Data":"bec1463378703b821265be5078ed722490921170871beb0bc9055d8fa3322db1"}
Sep 15 10:18:56.996177 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:56.995772    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:18:57.014221 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:57.012783    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gcxvj" podStartSLOduration=1.435029726 podStartE2EDuration="3.012762379s" totalImagesPullingTime="1.577732653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:57.011637355 +0000 UTC m=+55.705793028" watchObservedRunningTime="2026-09-15 10:18:57.012762379 +0000 UTC m=+55.706918060"
Sep 15 10:18:57.999791 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:57.999748    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hwtxj" event={"ID":"146f0ea1-aecd-451f-b1f1-8bc242b439d8","Type":"ContainerStarted","Data":"3e02df469880e7836d9e69de9b2cf7596e8abb1e32d1ad1de925c7d8f8dcfc36"}
Sep 15 10:18:58.015772 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:58.015712    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hwtxj" podStartSLOduration=1.4999341880000001 podStartE2EDuration="4.015695843s" totalImagesPullingTime="2.515761655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:18:58.014477805 +0000 UTC m=+56.708633487" watchObservedRunningTime="2026-09-15 10:18:58.015695843 +0000 UTC m=+56.709851542"
Sep 15 10:18:59.925979 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:18:59.925950    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nnbj8"
Sep 15 10:19:07.001892 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.001856    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gcxvj"
Sep 15 10:19:07.516495 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.516457    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kq6\" (UniqueName: \"kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6\") pod \"network-check-target-dszpz\" (UID: \"aab62d15-9d50-4842-9f39-0ed41176b31a\") " pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:19:07.516495 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.516500    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:19:07.516719 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.516522    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:19:07.519012 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.518995    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 10:19:07.520045 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.520028    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 10:19:07.520099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.520042    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 10:19:07.529367 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.529256    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 10:19:07.529507 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.529437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6992d04d-f1f5-4521-a0da-27555d562676-metrics-certs\") pod \"network-metrics-daemon-bftt2\" (UID: \"6992d04d-f1f5-4521-a0da-27555d562676\") " pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:19:07.529597 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.529574    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/992c0c97-0af1-4197-82ca-e4c193a935f6-original-pull-secret\") pod \"global-pull-secret-syncer-txhqg\" (UID: \"992c0c97-0af1-4197-82ca-e4c193a935f6\") " pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:19:07.539140 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.539109    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9kq6\" (UniqueName: \"kubernetes.io/projected/aab62d15-9d50-4842-9f39-0ed41176b31a-kube-api-access-x9kq6\") pod \"network-check-target-dszpz\" (UID: \"aab62d15-9d50-4842-9f39-0ed41176b31a\") " pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:19:07.691487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.691460    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lrdhd\""
Sep 15 10:19:07.695691 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.695672    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txhqg"
Sep 15 10:19:07.699153 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.699132    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bftt2"
Sep 15 10:19:07.702624 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.702607    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-88p68\""
Sep 15 10:19:07.711478 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.711453    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:19:07.942545 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.942509    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4qg4r"]
Sep 15 10:19:07.944274 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.944258    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:07.946329 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.946306    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 10:19:07.946530 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.946509    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 10:19:07.946643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.946625    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 10:19:07.946712 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.946655    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 10:19:07.947380 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.947365    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fxj4z\""
Sep 15 10:19:07.953785 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.953762    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4qg4r"]
Sep 15 10:19:07.977420 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.977398    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-txhqg"]
Sep 15 10:19:07.980748 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:19:07.980727    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992c0c97_0af1_4197_82ca_e4c193a935f6.slice/crio-0a60895c9332e0cb20b0f6d606b3f6ce61c1f36624a81aa8fd5554f65f49c24b WatchSource:0}: Error finding container 0a60895c9332e0cb20b0f6d606b3f6ce61c1f36624a81aa8fd5554f65f49c24b: Status 404 returned error can't find the container with id 0a60895c9332e0cb20b0f6d606b3f6ce61c1f36624a81aa8fd5554f65f49c24b
Sep 15 10:19:07.984467 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.984446    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bftt2"]
Sep 15 10:19:07.987215 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:19:07.987193    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6992d04d_f1f5_4521_a0da_27555d562676.slice/crio-f4c2c0e1bec13000ab9381aea697c9ecef7ae80e1241b1b6a68820d222eca765 WatchSource:0}: Error finding container f4c2c0e1bec13000ab9381aea697c9ecef7ae80e1241b1b6a68820d222eca765: Status 404 returned error can't find the container with id f4c2c0e1bec13000ab9381aea697c9ecef7ae80e1241b1b6a68820d222eca765
Sep 15 10:19:07.994607 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:07.994583    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dszpz"]
Sep 15 10:19:07.997056 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:19:07.997033    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab62d15_9d50_4842_9f39_0ed41176b31a.slice/crio-cd28bfe87ad7a998cd61bf1dfb0242785d784ce1e2643991e978cf35c287ec2c WatchSource:0}: Error finding container cd28bfe87ad7a998cd61bf1dfb0242785d784ce1e2643991e978cf35c287ec2c: Status 404 returned error can't find the container with id cd28bfe87ad7a998cd61bf1dfb0242785d784ce1e2643991e978cf35c287ec2c
Sep 15 10:19:08.020271 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.020246    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-data-volume\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.020614 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.020283    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.020614 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.020326    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qf7b\" (UniqueName: \"kubernetes.io/projected/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-api-access-2qf7b\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.020614 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.020362    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.020614 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.020427    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-crio-socket\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.024000 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.023940    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dszpz" event={"ID":"aab62d15-9d50-4842-9f39-0ed41176b31a","Type":"ContainerStarted","Data":"cd28bfe87ad7a998cd61bf1dfb0242785d784ce1e2643991e978cf35c287ec2c"}
Sep 15 10:19:08.025001 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.024978    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-txhqg" event={"ID":"992c0c97-0af1-4197-82ca-e4c193a935f6","Type":"ContainerStarted","Data":"0a60895c9332e0cb20b0f6d606b3f6ce61c1f36624a81aa8fd5554f65f49c24b"}
Sep 15 10:19:08.026031 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.026010    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bftt2" event={"ID":"6992d04d-f1f5-4521-a0da-27555d562676","Type":"ContainerStarted","Data":"f4c2c0e1bec13000ab9381aea697c9ecef7ae80e1241b1b6a68820d222eca765"}
Sep 15 10:19:08.121374 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.121339    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.121519 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.121382    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-crio-socket\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.121519 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.121455    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-crio-socket\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.121519 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.121476    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-data-volume\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.121519 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.121504    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.121714 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.121532    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qf7b\" (UniqueName: \"kubernetes.io/projected/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-api-access-2qf7b\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.121714 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:08.121482    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:08.121714 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:08.121595    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 10:19:08.121714 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:08.121624    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls podName:89c8a64a-26f9-45b5-a4a8-c1d06fcffad1 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:08.621597587 +0000 UTC m=+67.315753260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4qg4r" (UID: "89c8a64a-26f9-45b5-a4a8-c1d06fcffad1") : secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:08.121714 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:08.121661    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm podName:89c8a64a-26f9-45b5-a4a8-c1d06fcffad1 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:08.621645613 +0000 UTC m=+67.315801284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4qg4r" (UID: "89c8a64a-26f9-45b5-a4a8-c1d06fcffad1") : configmap "kube-rbac-proxy" not found
Sep 15 10:19:08.121933 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.121824    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-data-volume\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.129363 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.129342    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qf7b\" (UniqueName: \"kubernetes.io/projected/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-api-access-2qf7b\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.320310 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.320226    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-fm6hc"]
Sep 15 10:19:08.322456 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.322437    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-fm6hc"
Sep 15 10:19:08.324954 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.324916    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 10:19:08.324954 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.324918    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-t8wcw\""
Sep 15 10:19:08.324954 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.324918    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 10:19:08.333049 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.332998    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-fm6hc"]
Sep 15 10:19:08.423475 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.423438    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bnnb\" (UniqueName: \"kubernetes.io/projected/8f151224-d92a-49a4-90e7-f4f2448dda7a-kube-api-access-8bnnb\") pod \"downloads-76f99584f4-fm6hc\" (UID: \"8f151224-d92a-49a4-90e7-f4f2448dda7a\") " pod="openshift-console/downloads-76f99584f4-fm6hc"
Sep 15 10:19:08.524153 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.523919    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bnnb\" (UniqueName: \"kubernetes.io/projected/8f151224-d92a-49a4-90e7-f4f2448dda7a-kube-api-access-8bnnb\") pod \"downloads-76f99584f4-fm6hc\" (UID: \"8f151224-d92a-49a4-90e7-f4f2448dda7a\") " pod="openshift-console/downloads-76f99584f4-fm6hc"
Sep 15 10:19:08.532279 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.532249    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bnnb\" (UniqueName: \"kubernetes.io/projected/8f151224-d92a-49a4-90e7-f4f2448dda7a-kube-api-access-8bnnb\") pod \"downloads-76f99584f4-fm6hc\" (UID: \"8f151224-d92a-49a4-90e7-f4f2448dda7a\") " pod="openshift-console/downloads-76f99584f4-fm6hc"
Sep 15 10:19:08.571048 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.570925    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gcxvj_42c7db80-61f8-46e5-8458-3db6c66a56c3/dns/0.log"
Sep 15 10:19:08.624658 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.624590    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.624658 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.624658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:08.624935 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:08.624724    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 10:19:08.624935 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:08.624789    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:08.624935 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:08.624811    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm podName:89c8a64a-26f9-45b5-a4a8-c1d06fcffad1 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:09.624774029 +0000 UTC m=+68.318929706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4qg4r" (UID: "89c8a64a-26f9-45b5-a4a8-c1d06fcffad1") : configmap "kube-rbac-proxy" not found
Sep 15 10:19:08.624935 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:08.624885    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls podName:89c8a64a-26f9-45b5-a4a8-c1d06fcffad1 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:09.624868215 +0000 UTC m=+68.319023876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4qg4r" (UID: "89c8a64a-26f9-45b5-a4a8-c1d06fcffad1") : secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:08.631588 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.631562    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-fm6hc"
Sep 15 10:19:08.755915 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.755868    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gcxvj_42c7db80-61f8-46e5-8458-3db6c66a56c3/kube-rbac-proxy/0.log"
Sep 15 10:19:08.934533 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:08.934443    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-fm6hc"]
Sep 15 10:19:09.128187 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:19:09.128140    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f151224_d92a_49a4_90e7_f4f2448dda7a.slice/crio-37a77530f2c17483ccd95d784b03523db3cf174da9ecc714b4855d81c739cbeb WatchSource:0}: Error finding container 37a77530f2c17483ccd95d784b03523db3cf174da9ecc714b4855d81c739cbeb: Status 404 returned error can't find the container with id 37a77530f2c17483ccd95d784b03523db3cf174da9ecc714b4855d81c739cbeb
Sep 15 10:19:09.633497 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:09.633464    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:09.633683 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:09.633529    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:09.633683 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:09.633613    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 10:19:09.633683 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:09.633635    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:09.633847 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:09.633696    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm podName:89c8a64a-26f9-45b5-a4a8-c1d06fcffad1 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:11.633675276 +0000 UTC m=+70.327830950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4qg4r" (UID: "89c8a64a-26f9-45b5-a4a8-c1d06fcffad1") : configmap "kube-rbac-proxy" not found
Sep 15 10:19:09.633847 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:09.633718    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls podName:89c8a64a-26f9-45b5-a4a8-c1d06fcffad1 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:11.633708845 +0000 UTC m=+70.327864518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4qg4r" (UID: "89c8a64a-26f9-45b5-a4a8-c1d06fcffad1") : secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:09.953737 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:09.953646    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q4zj5_b1e9ea1a-0387-4294-a760-df3980219768/dns-node-resolver/0.log"
Sep 15 10:19:10.037268 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:10.037227    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-fm6hc" event={"ID":"8f151224-d92a-49a4-90e7-f4f2448dda7a","Type":"ContainerStarted","Data":"37a77530f2c17483ccd95d784b03523db3cf174da9ecc714b4855d81c739cbeb"}
Sep 15 10:19:10.039854 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:10.039793    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bftt2" event={"ID":"6992d04d-f1f5-4521-a0da-27555d562676","Type":"ContainerStarted","Data":"9a1e1bad44db4c8d912e3d5ed821c94a97fac1820f096de01031d792abc5c330"}
Sep 15 10:19:10.040011 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:10.039880    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bftt2" event={"ID":"6992d04d-f1f5-4521-a0da-27555d562676","Type":"ContainerStarted","Data":"fe5de0bbd6cd3071b50a388d2fbdef196e0d3d845a2e11869f935b5104b1cb2f"}
Sep 15 10:19:10.058970 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:10.058250    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bftt2" podStartSLOduration=67.877619469 podStartE2EDuration="1m9.058228716s" totalImagesPullingTime="1.180609247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:19:10.056750308 +0000 UTC m=+68.750905989" watchObservedRunningTime="2026-09-15 10:19:10.058228716 +0000 UTC m=+68.752384396"
Sep 15 10:19:10.557012 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:10.556920    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79dd5bfd77-hrtl6_f1c396c5-e224-478f-a09d-6857eee1ae0e/registry/0.log"
Sep 15 10:19:10.953027 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:10.952793    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7vfnf_72c44564-2d7f-47fd-8567-d8793ed885bd/node-ca/0.log"
Sep 15 10:19:11.648412 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:11.648356    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:11.648909 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:11.648431    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:11.648909 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:11.648497    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 10:19:11.648909 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:11.648547    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:11.648909 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:11.648571    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm podName:89c8a64a-26f9-45b5-a4a8-c1d06fcffad1 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:15.648553222 +0000 UTC m=+74.342708904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4qg4r" (UID: "89c8a64a-26f9-45b5-a4a8-c1d06fcffad1") : configmap "kube-rbac-proxy" not found
Sep 15 10:19:11.648909 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:11.648592    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls podName:89c8a64a-26f9-45b5-a4a8-c1d06fcffad1 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:15.648579887 +0000 UTC m=+74.342735562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4qg4r" (UID: "89c8a64a-26f9-45b5-a4a8-c1d06fcffad1") : secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:11.753301 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:11.753249    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hwtxj_146f0ea1-aecd-451f-b1f1-8bc242b439d8/serve-healthcheck-canary/0.log"
Sep 15 10:19:14.058052 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:14.058017    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dszpz" event={"ID":"aab62d15-9d50-4842-9f39-0ed41176b31a","Type":"ContainerStarted","Data":"77423747adba328abdb11cdc03ac604b266a98b682274e733be6ea8a66e62550"}
Sep 15 10:19:14.058557 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:14.058098    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:19:14.060171 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:14.060143    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-txhqg" event={"ID":"992c0c97-0af1-4197-82ca-e4c193a935f6","Type":"ContainerStarted","Data":"94fd731ca6c8ffbd9d65baa5bd21621b47040459af391a6c8aca2309126d2106"}
Sep 15 10:19:14.073851 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:14.073777    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dszpz" podStartSLOduration=67.65964026 podStartE2EDuration="1m13.073758694s" totalImagesPullingTime="5.414118434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:19:14.073494738 +0000 UTC m=+72.767650449" watchObservedRunningTime="2026-09-15 10:19:14.073758694 +0000 UTC m=+72.767914374"
Sep 15 10:19:14.087494 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:14.087440    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-txhqg" podStartSLOduration=67.655660466 podStartE2EDuration="1m13.08742499s" totalImagesPullingTime="5.431764524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:19:14.087291449 +0000 UTC m=+72.781447133" watchObservedRunningTime="2026-09-15 10:19:14.08742499 +0000 UTC m=+72.781580667"
Sep 15 10:19:14.394988 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:14.394903    2584 patch_prober.go:28] interesting pod/image-registry-79dd5bfd77-hrtl6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 10:19:14.394988 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:14.394969    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6" podUID="f1c396c5-e224-478f-a09d-6857eee1ae0e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 10:19:15.681275 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:15.681230    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:15.681741 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:15.681308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:15.681741 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:15.681409    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:15.681741 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:15.681424    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 10:19:15.681741 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:15.681493    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls podName:89c8a64a-26f9-45b5-a4a8-c1d06fcffad1 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:23.681471327 +0000 UTC m=+82.375627001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4qg4r" (UID: "89c8a64a-26f9-45b5-a4a8-c1d06fcffad1") : secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:15.681741 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:15.681514    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm podName:89c8a64a-26f9-45b5-a4a8-c1d06fcffad1 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:23.681505302 +0000 UTC m=+82.375660960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4qg4r" (UID: "89c8a64a-26f9-45b5-a4a8-c1d06fcffad1") : configmap "kube-rbac-proxy" not found
Sep 15 10:19:15.994372 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:15.994341    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79dd5bfd77-hrtl6"
Sep 15 10:19:19.974521 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:19.974488    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"]
Sep 15 10:19:19.978931 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:19.978899    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:19.981342 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:19.981320    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 10:19:19.981658 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:19.981638    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9k9pz\""
Sep 15 10:19:19.981829 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:19.981788    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 10:19:19.982724 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:19.982701    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 10:19:19.982724 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:19.982715    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 10:19:19.982910 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:19.982706    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 10:19:19.991396 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:19.991367    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"]
Sep 15 10:19:20.001237 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.001209    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"]
Sep 15 10:19:20.004616 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.004594    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6bzth"]
Sep 15 10:19:20.004757 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.004736    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.007539 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.007518    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 10:19:20.007700 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.007555    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 10:19:20.007700 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.007580    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qmvrz\""
Sep 15 10:19:20.007865 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.007713    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 10:19:20.008249 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.008232    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.010569 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.010546    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 10:19:20.010664 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.010588    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 10:19:20.010852 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.010820    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vpk4z\""
Sep 15 10:19:20.010930 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.010825    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 10:19:20.016548 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.016528    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"]
Sep 15 10:19:20.020928 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.020904    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.021017 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.020943    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rpgw\" (UniqueName: \"kubernetes.io/projected/0ecc33dd-55c1-491c-918a-5d52916e5535-kube-api-access-2rpgw\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.021017 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.020988    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ecc33dd-55c1-491c-918a-5d52916e5535-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.021106 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.021028    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.122136 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122101    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.122316 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.122316 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122182    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rpgw\" (UniqueName: \"kubernetes.io/projected/0ecc33dd-55c1-491c-918a-5d52916e5535-kube-api-access-2rpgw\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.122316 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122206    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.122316 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122237    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-wtmp\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.122316 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122272    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vjm\" (UniqueName: \"kubernetes.io/projected/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-kube-api-access-g9vjm\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.122316 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122292    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.122582 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:20.122313    2584 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 15 10:19:20.122582 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:20.122385    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-tls podName:0ecc33dd-55c1-491c-918a-5d52916e5535 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:20.622364241 +0000 UTC m=+79.316519904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-k95kh" (UID: "0ecc33dd-55c1-491c-918a-5d52916e5535") : secret "openshift-state-metrics-tls" not found
Sep 15 10:19:20.122582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122317    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.122582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122440    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-root\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.122582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122480    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-accelerators-collector-config\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.122582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122519    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-tls\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.122582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122541    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.122582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122570    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.122965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122603    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ecc33dd-55c1-491c-918a-5d52916e5535-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.122965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-metrics-client-ca\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.122965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.122965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122698    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ht7\" (UniqueName: \"kubernetes.io/projected/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-api-access-f6ht7\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.122965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122729    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-sys\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.122965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.122753    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-textfile\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.123421 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.123395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ecc33dd-55c1-491c-918a-5d52916e5535-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.125336 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.125314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.142236 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.142207    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rpgw\" (UniqueName: \"kubernetes.io/projected/0ecc33dd-55c1-491c-918a-5d52916e5535-kube-api-access-2rpgw\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.224137 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224097    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-tls\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.224327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.224327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224178    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.224327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224214    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-metrics-client-ca\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.224327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224265    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ht7\" (UniqueName: \"kubernetes.io/projected/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-api-access-f6ht7\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.224327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224301    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-sys\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.224327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224326    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-textfile\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.224721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224407    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-sys\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.224721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224405    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.224721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224472    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.224721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224508    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-wtmp\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.224721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224548    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vjm\" (UniqueName: \"kubernetes.io/projected/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-kube-api-access-g9vjm\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.224721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224577    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.224721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224606    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.224721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224633    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-root\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.224721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-accelerators-collector-config\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.225190 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.224863    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.225190 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.225006    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-metrics-client-ca\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.225190 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.225025    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.225190 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.225082    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.225190 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.225170    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-wtmp\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.225441 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.225219    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-accelerators-collector-config\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.225441 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.225281    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-textfile\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.225441 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.225283    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-root\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.226753 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.226730    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-tls\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.227751 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.227659    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.227751 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.227683    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.228570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.228550    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.234266 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.234247    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vjm\" (UniqueName: \"kubernetes.io/projected/796cc65d-7664-4e9a-ba6f-7c0da2ac0b64-kube-api-access-g9vjm\") pod \"node-exporter-6bzth\" (UID: \"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64\") " pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.237594 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.237575    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ht7\" (UniqueName: \"kubernetes.io/projected/d708ecc6-3d6a-4921-a47d-3a17f0de0b1a-kube-api-access-f6ht7\") pod \"kube-state-metrics-59fbdd77d-w997q\" (UID: \"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.316741 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.316705    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"
Sep 15 10:19:20.324295 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.324270    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6bzth"
Sep 15 10:19:20.628561 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:20.628519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:20.628758 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:20.628667    2584 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 15 10:19:20.628851 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:20.628761    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-tls podName:0ecc33dd-55c1-491c-918a-5d52916e5535 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:21.628739451 +0000 UTC m=+80.322895187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-k95kh" (UID: "0ecc33dd-55c1-491c-918a-5d52916e5535") : secret "openshift-state-metrics-tls" not found
Sep 15 10:19:21.047124 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.047087    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 10:19:21.050883 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.050861    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.053514 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.053490    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 10:19:21.053641 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.053536    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 10:19:21.053641 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.053592    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 10:19:21.053758 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.053715    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 10:19:21.053758 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.053718    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 10:19:21.053900 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.053780    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 10:19:21.053900 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.053780    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s2b8b\""
Sep 15 10:19:21.054095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.054078    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 10:19:21.054095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.054090    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 10:19:21.054518 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.054497    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 10:19:21.066147 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.066118    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 10:19:21.132125 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132080    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-out\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132125 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132122    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132144    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvz5s\" (UniqueName: \"kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-kube-api-access-bvz5s\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132190    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132310    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132346    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-volume\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132496 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132366    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132496 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132393    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132496 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132439    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132496 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132474    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-web-config\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132657 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132503    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132657 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132531    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.132657 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.132556    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.233615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.233574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.233825 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.233630    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-volume\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.233825 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.233656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.233825 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.233689    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.233825 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.233727    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.234062 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.233843    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-web-config\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.234062 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.233888    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.234062 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.233917    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.234062 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.233947    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.234062 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.233997    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-out\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.234062 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:21.234007    2584 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 15 10:19:21.234062 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.234020    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.234062 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.234042    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvz5s\" (UniqueName: \"kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-kube-api-access-bvz5s\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.234452 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:21.234080    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls podName:5118ee90-ec86-4e54-9dfb-d4c6d8ec2700 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:21.734061537 +0000 UTC m=+80.428217209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700") : secret "alertmanager-main-tls" not found
Sep 15 10:19:21.234452 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.234110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.234452 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:21.234433    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-trusted-ca-bundle podName:5118ee90-ec86-4e54-9dfb-d4c6d8ec2700 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:21.734414477 +0000 UTC m=+80.428570149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700") : configmap references non-existent config key: ca-bundle.crt
Sep 15 10:19:21.234614 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.234538    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.235496 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.235141    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.237314 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.237287    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.237925 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.237877    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.237925 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.237900    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-volume\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.238343 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.238124    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-out\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.238480 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.238370    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.238636 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.238615    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-web-config\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.238886 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.238777    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.239292 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.239273    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.242608 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.242589    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvz5s\" (UniqueName: \"kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-kube-api-access-bvz5s\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.638017 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.637978    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:21.640864 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.640834    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ecc33dd-55c1-491c-918a-5d52916e5535-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-k95kh\" (UID: \"0ecc33dd-55c1-491c-918a-5d52916e5535\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:21.739083 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.739047    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.739243 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.739091    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.739243 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:21.739217    2584 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 15 10:19:21.739370 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:21.739294    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls podName:5118ee90-ec86-4e54-9dfb-d4c6d8ec2700 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:22.739273808 +0000 UTC m=+81.433429485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700") : secret "alertmanager-main-tls" not found
Sep 15 10:19:21.740030 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.739998    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:21.789444 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:21.789409    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"
Sep 15 10:19:22.748718 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:22.748674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:22.751476 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:22.751451    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:22.861866 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:22.861828    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:19:23.757062 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:23.757021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:23.757503 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:23.757084    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:23.757503 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:23.757195    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:23.757503 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:19:23.757267    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls podName:89c8a64a-26f9-45b5-a4a8-c1d06fcffad1 nodeName:}" failed. No retries permitted until 2026-09-15 10:19:39.757249922 +0000 UTC m=+98.451405602 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4qg4r" (UID: "89c8a64a-26f9-45b5-a4a8-c1d06fcffad1") : secret "insights-runtime-extractor-tls" not found
Sep 15 10:19:23.757620 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:23.757597    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:24.427951 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.427916    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"]
Sep 15 10:19:24.431134 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.431112    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.434614 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.434596    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 10:19:24.434857 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.434834    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 10:19:24.434947 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.434878    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7m82452cgreea\""
Sep 15 10:19:24.435005 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.434944    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 10:19:24.435056 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.435004    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4k9ls\""
Sep 15 10:19:24.435056 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.435039    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 10:19:24.439820 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.439786    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"]
Sep 15 10:19:24.566122 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.566089    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/83738ffd-b578-426f-a6a4-b66e88b01b7f-client-ca-bundle\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.566122 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.566128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/83738ffd-b578-426f-a6a4-b66e88b01b7f-audit-log\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.566376 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.566157    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57z4t\" (UniqueName: \"kubernetes.io/projected/83738ffd-b578-426f-a6a4-b66e88b01b7f-kube-api-access-57z4t\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.566376 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.566227    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/83738ffd-b578-426f-a6a4-b66e88b01b7f-secret-metrics-server-tls\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.566376 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.566327    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/83738ffd-b578-426f-a6a4-b66e88b01b7f-metrics-server-audit-profiles\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.566502 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.566386    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/83738ffd-b578-426f-a6a4-b66e88b01b7f-secret-metrics-server-client-certs\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.566502 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.566418    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83738ffd-b578-426f-a6a4-b66e88b01b7f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.667761 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.667723    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/83738ffd-b578-426f-a6a4-b66e88b01b7f-secret-metrics-server-tls\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.667968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.667781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/83738ffd-b578-426f-a6a4-b66e88b01b7f-metrics-server-audit-profiles\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.667968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.667836    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/83738ffd-b578-426f-a6a4-b66e88b01b7f-secret-metrics-server-client-certs\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.667968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.667867    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83738ffd-b578-426f-a6a4-b66e88b01b7f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.667968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.667902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/83738ffd-b578-426f-a6a4-b66e88b01b7f-client-ca-bundle\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.667968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.667930    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/83738ffd-b578-426f-a6a4-b66e88b01b7f-audit-log\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.667968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.667957    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57z4t\" (UniqueName: \"kubernetes.io/projected/83738ffd-b578-426f-a6a4-b66e88b01b7f-kube-api-access-57z4t\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.668652 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.668617    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/83738ffd-b578-426f-a6a4-b66e88b01b7f-audit-log\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.668781 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.668678    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83738ffd-b578-426f-a6a4-b66e88b01b7f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.668961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.668922    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/83738ffd-b578-426f-a6a4-b66e88b01b7f-metrics-server-audit-profiles\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.670565 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.670537    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/83738ffd-b578-426f-a6a4-b66e88b01b7f-secret-metrics-server-client-certs\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.670856 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.670834    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/83738ffd-b578-426f-a6a4-b66e88b01b7f-client-ca-bundle\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.670986 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.670964    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/83738ffd-b578-426f-a6a4-b66e88b01b7f-secret-metrics-server-tls\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.675776 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.675753    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57z4t\" (UniqueName: \"kubernetes.io/projected/83738ffd-b578-426f-a6a4-b66e88b01b7f-kube-api-access-57z4t\") pod \"metrics-server-5fc745c9c8-5v7b4\" (UID: \"83738ffd-b578-426f-a6a4-b66e88b01b7f\") " pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:24.743175 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:24.743139    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:25.655348 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:19:25.655307    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796cc65d_7664_4e9a_ba6f_7c0da2ac0b64.slice/crio-70e4e227999c803641e0f15ec84c526bc36d3a67c1f20d3c543e6f72fb043c06 WatchSource:0}: Error finding container 70e4e227999c803641e0f15ec84c526bc36d3a67c1f20d3c543e6f72fb043c06: Status 404 returned error can't find the container with id 70e4e227999c803641e0f15ec84c526bc36d3a67c1f20d3c543e6f72fb043c06
Sep 15 10:19:25.938579 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:25.938202    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-w997q"]
Sep 15 10:19:25.940628 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:25.940492    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"]
Sep 15 10:19:25.941010 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:25.940985    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh"]
Sep 15 10:19:25.942540 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:25.942462    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 10:19:25.943059 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:19:25.943029    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecc33dd_55c1_491c_918a_5d52916e5535.slice/crio-81e3e6418ca8b8b8a9bf96b523573a585b7206f86292896fe6e91ad901149d8d WatchSource:0}: Error finding container 81e3e6418ca8b8b8a9bf96b523573a585b7206f86292896fe6e91ad901149d8d: Status 404 returned error can't find the container with id 81e3e6418ca8b8b8a9bf96b523573a585b7206f86292896fe6e91ad901149d8d
Sep 15 10:19:25.943551 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:19:25.943522    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83738ffd_b578_426f_a6a4_b66e88b01b7f.slice/crio-230ba57e0ed7ea3836bdf7a385bd4082344aa1442598ace3d332a3ede810a849 WatchSource:0}: Error finding container 230ba57e0ed7ea3836bdf7a385bd4082344aa1442598ace3d332a3ede810a849: Status 404 returned error can't find the container with id 230ba57e0ed7ea3836bdf7a385bd4082344aa1442598ace3d332a3ede810a849
Sep 15 10:19:25.944470 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:19:25.944442    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd708ecc6_3d6a_4921_a47d_3a17f0de0b1a.slice/crio-fa0732ffe96bb59a95e7e2916e6044259c9cf760335401d62866b58340095992 WatchSource:0}: Error finding container fa0732ffe96bb59a95e7e2916e6044259c9cf760335401d62866b58340095992: Status 404 returned error can't find the container with id fa0732ffe96bb59a95e7e2916e6044259c9cf760335401d62866b58340095992
Sep 15 10:19:25.947036 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:19:25.946988    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5118ee90_ec86_4e54_9dfb_d4c6d8ec2700.slice/crio-805220e956565bb8431e87ccb5ff9cda9eaa31027c113a3f553af772bbc82a6a WatchSource:0}: Error finding container 805220e956565bb8431e87ccb5ff9cda9eaa31027c113a3f553af772bbc82a6a: Status 404 returned error can't find the container with id 805220e956565bb8431e87ccb5ff9cda9eaa31027c113a3f553af772bbc82a6a
Sep 15 10:19:26.117482 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.117428    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-fm6hc" event={"ID":"8f151224-d92a-49a4-90e7-f4f2448dda7a","Type":"ContainerStarted","Data":"54495d0723f5f2b07da9afdcc06d48eeacfeb7a6604416dc96f194497789558a"}
Sep 15 10:19:26.120046 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.117905    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-fm6hc"
Sep 15 10:19:26.120448 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.120230    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerStarted","Data":"805220e956565bb8431e87ccb5ff9cda9eaa31027c113a3f553af772bbc82a6a"}
Sep 15 10:19:26.122862 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.122831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4" event={"ID":"83738ffd-b578-426f-a6a4-b66e88b01b7f","Type":"ContainerStarted","Data":"230ba57e0ed7ea3836bdf7a385bd4082344aa1442598ace3d332a3ede810a849"}
Sep 15 10:19:26.124880 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.124829    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q" event={"ID":"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a","Type":"ContainerStarted","Data":"fa0732ffe96bb59a95e7e2916e6044259c9cf760335401d62866b58340095992"}
Sep 15 10:19:26.126370 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.126339    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bzth" event={"ID":"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64","Type":"ContainerStarted","Data":"70e4e227999c803641e0f15ec84c526bc36d3a67c1f20d3c543e6f72fb043c06"}
Sep 15 10:19:26.130230 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.130190    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-fm6hc"
Sep 15 10:19:26.130333 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.130256    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh" event={"ID":"0ecc33dd-55c1-491c-918a-5d52916e5535","Type":"ContainerStarted","Data":"b43960c6f791b21cc2d938f537c2904181cd963a2b95ba5e3eb1ae96d5d66af7"}
Sep 15 10:19:26.130333 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.130295    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh" event={"ID":"0ecc33dd-55c1-491c-918a-5d52916e5535","Type":"ContainerStarted","Data":"ccd86da58948677c384b191091a1237e748c41bcfac402b19e7a056b7bebb762"}
Sep 15 10:19:26.130333 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.130321    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh" event={"ID":"0ecc33dd-55c1-491c-918a-5d52916e5535","Type":"ContainerStarted","Data":"81e3e6418ca8b8b8a9bf96b523573a585b7206f86292896fe6e91ad901149d8d"}
Sep 15 10:19:26.138475 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.138384    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-fm6hc" podStartSLOduration=1.507197702 podStartE2EDuration="18.138344651s" totalImagesPullingTime="16.631146949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:19:26.137323459 +0000 UTC m=+84.831479140" watchObservedRunningTime="2026-09-15 10:19:26.138344651 +0000 UTC m=+84.832500331"
Sep 15 10:19:26.161303 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.161269    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 10:19:26.165661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.165627    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.169334 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.169314    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 10:19:26.169584 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.169561    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 10:19:26.170051 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.170031    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 10:19:26.170163 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.170138    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 10:19:26.170472 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.170456    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 10:19:26.170745 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.170727    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 10:19:26.171095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.171078    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m7zwv\""
Sep 15 10:19:26.171557 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.171539    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 10:19:26.171915 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.171897    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bd4g57ichahks\""
Sep 15 10:19:26.172136 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.172114    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 10:19:26.172341 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.172322    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 10:19:26.173155 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.173129    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 10:19:26.173347 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.171993    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 10:19:26.173810 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.173771    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 10:19:26.175507 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.175486    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 10:19:26.178969 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.178787    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 10:19:26.187430 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.187345    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 10:19:26.285101 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwswn\" (UniqueName: \"kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-kube-api-access-mwswn\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285320 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285113    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285320 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285148    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285320 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285236    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285320 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285294    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285526 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285324    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285526 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285352    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285526 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285391    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285526 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285409    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285526 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285426    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285526 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285464    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285526 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285493    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285526 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285522    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285939 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285939 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285582    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285939 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285609    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285939 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285649    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285939 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285677    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285939 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.285939 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.285748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.386545 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.386500    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.386725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.386550    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.386792 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.386767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.386913 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.386886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387044 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.386942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387044 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.386967    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387044 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387008    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387220 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387054    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387220 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387083    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387321 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387173    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387321 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387242    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387321 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387252    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387478 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387349    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387478 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387376    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387478 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387421    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387630 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387565    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387714 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387692    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387774 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387843 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387776    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387843 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387824    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwswn\" (UniqueName: \"kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-kube-api-access-mwswn\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387849    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.387962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.387880    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.390359 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.390332    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.390452 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.390375    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.390527 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.390478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.391640 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.390726    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.391640 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.391359    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.391640 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.391599    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.393669 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.393642    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.394016 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.393974    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.394395 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.394372    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.394862 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.394819    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.395305 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.395282    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.395428 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.395405    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.395787 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.395763    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.398262 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.398151    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.398262 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.398216    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.399726 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.399682    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.400611 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.400555    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwswn\" (UniqueName: \"kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-kube-api-access-mwswn\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.401173 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.401079    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:26.485468 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:26.485191    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:27.180739 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:27.180661    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 10:19:27.728383 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:19:27.728314    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb81b57_aea5_4ccb_ab60_c34a9ca988e1.slice/crio-f60d15fde03288603bd5b00342c1b95532cb667bd39fc6b3151b1452d6d8bd0a WatchSource:0}: Error finding container f60d15fde03288603bd5b00342c1b95532cb667bd39fc6b3151b1452d6d8bd0a: Status 404 returned error can't find the container with id f60d15fde03288603bd5b00342c1b95532cb667bd39fc6b3151b1452d6d8bd0a
Sep 15 10:19:28.140662 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:28.140564    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerStarted","Data":"f60d15fde03288603bd5b00342c1b95532cb667bd39fc6b3151b1452d6d8bd0a"}
Sep 15 10:19:30.156846 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:30.153932    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bzth" event={"ID":"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64","Type":"ContainerStarted","Data":"549bfbb64f87d6291875c2c7ca9e68dcb8eab066a2a2e01572f697b037989fbe"}
Sep 15 10:19:30.162533 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:30.162392    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerStarted","Data":"598ad50015967e3444ed6b51615ad72fdbc3ce6819429cfffda4dc8e33920298"}
Sep 15 10:19:30.165887 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:30.165837    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4" event={"ID":"83738ffd-b578-426f-a6a4-b66e88b01b7f","Type":"ContainerStarted","Data":"b4205c37b4416acd69a06e6fc29084c281ae33613ace1f48980fc07279aa72a0"}
Sep 15 10:19:30.168120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:30.168093    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q" event={"ID":"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a","Type":"ContainerStarted","Data":"ba58ae37146c34d2e68d7828ff1ad0e1d0a69e8c5bd6e3115c7481af91fc210f"}
Sep 15 10:19:31.178764 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.178678    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh" event={"ID":"0ecc33dd-55c1-491c-918a-5d52916e5535","Type":"ContainerStarted","Data":"2264f788d5892db32f0fcbcce5918143a0153e53c9991a1de4b7f2be2cffdf6a"}
Sep 15 10:19:31.180615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.180582    2584 generic.go:356] "Generic (PLEG): container finished" podID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerID="598ad50015967e3444ed6b51615ad72fdbc3ce6819429cfffda4dc8e33920298" exitCode=0
Sep 15 10:19:31.180736 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.180668    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerDied","Data":"598ad50015967e3444ed6b51615ad72fdbc3ce6819429cfffda4dc8e33920298"}
Sep 15 10:19:31.183314 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.183282    2584 generic.go:356] "Generic (PLEG): container finished" podID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerID="0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19" exitCode=0
Sep 15 10:19:31.183399 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.183357    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerDied","Data":"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"}
Sep 15 10:19:31.186730 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.186508    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q" event={"ID":"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a","Type":"ContainerStarted","Data":"5428c9149ed6ea4a7f5129939941d7fb175c87e9e854534c1640b84b65e8f276"}
Sep 15 10:19:31.186730 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.186568    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q" event={"ID":"d708ecc6-3d6a-4921-a47d-3a17f0de0b1a","Type":"ContainerStarted","Data":"1f8a031f0679acb14e9a39740f59c75191db6911bff2b1fb3e8c790943f6edd3"}
Sep 15 10:19:31.188299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.188271    2584 generic.go:356] "Generic (PLEG): container finished" podID="796cc65d-7664-4e9a-ba6f-7c0da2ac0b64" containerID="549bfbb64f87d6291875c2c7ca9e68dcb8eab066a2a2e01572f697b037989fbe" exitCode=0
Sep 15 10:19:31.188380 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.188327    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bzth" event={"ID":"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64","Type":"ContainerDied","Data":"549bfbb64f87d6291875c2c7ca9e68dcb8eab066a2a2e01572f697b037989fbe"}
Sep 15 10:19:31.195143 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.195095    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-k95kh" podStartSLOduration=8.327841465 podStartE2EDuration="12.195081888s" totalImagesPullingTime="3.867240423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:19:31.193746292 +0000 UTC m=+89.887901973" watchObservedRunningTime="2026-09-15 10:19:31.195081888 +0000 UTC m=+89.889237568"
Sep 15 10:19:31.225532 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.225461    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w997q" podStartSLOduration=8.652684034 podStartE2EDuration="12.22544178s" totalImagesPullingTime="3.572757746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:19:31.224204931 +0000 UTC m=+89.918360612" watchObservedRunningTime="2026-09-15 10:19:31.22544178 +0000 UTC m=+89.919597462"
Sep 15 10:19:31.241467 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:31.241418    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4" podStartSLOduration=3.226046554 podStartE2EDuration="7.241396804s" totalImagesPullingTime="4.01535025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:19:31.240226936 +0000 UTC m=+89.934382619" watchObservedRunningTime="2026-09-15 10:19:31.241396804 +0000 UTC m=+89.935552490"
Sep 15 10:19:32.199336 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:32.198621    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bzth" event={"ID":"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64","Type":"ContainerStarted","Data":"760f3a826b5b2f1fbdfe758171090b20a8a1890934b15ce57debad25c1c7aeaa"}
Sep 15 10:19:32.199336 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:32.198685    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bzth" event={"ID":"796cc65d-7664-4e9a-ba6f-7c0da2ac0b64","Type":"ContainerStarted","Data":"fb38170c8c004777119231e45b1b043d3956e5ca47ae3feda4965d3dc0aeae06"}
Sep 15 10:19:32.219253 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:32.219189    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6bzth" podStartSLOduration=9.689307569 podStartE2EDuration="13.2191657s" totalImagesPullingTime="3.365175439s" totalInitContainerRuntime="164.682692ms" isStatefulPod=true podCreationTimestamp="2026-09-15 10:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:19:32.219128798 +0000 UTC m=+90.913284481" watchObservedRunningTime="2026-09-15 10:19:32.2191657 +0000 UTC m=+90.913321394"
Sep 15 10:19:36.221355 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:36.221269    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerStarted","Data":"9a3d760d2a61cbb89bb7a30495ed9246f480791d11a7f73945399d5b916d42bd"}
Sep 15 10:19:36.221355 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:36.221352    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerStarted","Data":"c90243e4d3ee38feff3555cbc91027ca6e0b144a38e6b2008b5f4aac69bde78f"}
Sep 15 10:19:36.221883 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:36.221389    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerStarted","Data":"ee04cfc9f56e3d37a7092f71413512636749651edacdb2acee78c2b236580585"}
Sep 15 10:19:36.221883 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:36.221425    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerStarted","Data":"7ef7eca43c175c3cbf5f78763939c1942c6e9e038f8c5ad389becb6398dc5919"}
Sep 15 10:19:36.221883 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:36.221465    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerStarted","Data":"8ffc96d412d3696c3cc28702df40523bba1bea358fdd50cfe244559038bce4df"}
Sep 15 10:19:36.223367 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:36.223342    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerStarted","Data":"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"}
Sep 15 10:19:36.223470 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:36.223395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerStarted","Data":"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"}
Sep 15 10:19:38.234118 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:38.234080    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerStarted","Data":"72d664ff0f83aec8ae578dbdf0e639ba20a90a75bc1809f1e2fb68ab324aaeeb"}
Sep 15 10:19:38.260908 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:38.260778    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.053283853 podStartE2EDuration="17.26075658s" totalImagesPullingTime="9.172234961s" totalInitContainerRuntime="35.237766ms" isStatefulPod=false podCreationTimestamp="2026-09-15 10:19:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:19:38.257720474 +0000 UTC m=+96.951876154" watchObservedRunningTime="2026-09-15 10:19:38.26075658 +0000 UTC m=+96.954912261"
Sep 15 10:19:39.241740 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:39.241700    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerStarted","Data":"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"}
Sep 15 10:19:39.242164 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:39.241815    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerStarted","Data":"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"}
Sep 15 10:19:39.242164 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:39.241852    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerStarted","Data":"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"}
Sep 15 10:19:39.242164 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:39.241877    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerStarted","Data":"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"}
Sep 15 10:19:39.270055 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:39.270000    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.094674131 podStartE2EDuration="13.269980918s" totalImagesPullingTime="9.128697768s" totalInitContainerRuntime="46.609019ms" isStatefulPod=false podCreationTimestamp="2026-09-15 10:19:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:19:39.26656411 +0000 UTC m=+97.960719792" watchObservedRunningTime="2026-09-15 10:19:39.269980918 +0000 UTC m=+97.964136598"
Sep 15 10:19:39.831599 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:39.831560    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:39.833954 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:39.833921    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89c8a64a-26f9-45b5-a4a8-c1d06fcffad1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qg4r\" (UID: \"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1\") " pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:40.053985 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:40.053941    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4qg4r"
Sep 15 10:19:40.353779 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:40.353702    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4qg4r"]
Sep 15 10:19:40.356942 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:19:40.356912    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c8a64a_26f9_45b5_a4a8_c1d06fcffad1.slice/crio-982c8f3af93adb57455c0cff42aa3387359d58e6b5c3935aee23e7e3f37ba36e WatchSource:0}: Error finding container 982c8f3af93adb57455c0cff42aa3387359d58e6b5c3935aee23e7e3f37ba36e: Status 404 returned error can't find the container with id 982c8f3af93adb57455c0cff42aa3387359d58e6b5c3935aee23e7e3f37ba36e
Sep 15 10:19:41.252066 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:41.252019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4qg4r" event={"ID":"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1","Type":"ContainerStarted","Data":"a9b6c43ed6752254a3a8373753d85778e24a731b216d34827d39cd6828b44773"}
Sep 15 10:19:41.252377 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:41.252089    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4qg4r" event={"ID":"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1","Type":"ContainerStarted","Data":"982c8f3af93adb57455c0cff42aa3387359d58e6b5c3935aee23e7e3f37ba36e"}
Sep 15 10:19:41.486057 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:41.486023    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:19:42.257477 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:42.257444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4qg4r" event={"ID":"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1","Type":"ContainerStarted","Data":"f7f9bd3b8a0906c69f98c95fea54545473479fb66411074107f4994ab8769f1e"}
Sep 15 10:19:44.268231 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:44.268187    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4qg4r" event={"ID":"89c8a64a-26f9-45b5-a4a8-c1d06fcffad1","Type":"ContainerStarted","Data":"1b1394b9adadf045113f40fc0e9b8fc35fd1aaece71b0db1db398a4431746469"}
Sep 15 10:19:44.286650 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:44.286596    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4qg4r" podStartSLOduration=33.962139784 podStartE2EDuration="37.28658105s" totalImagesPullingTime="3.324441266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 10:19:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:19:44.285384968 +0000 UTC m=+102.979540648" watchObservedRunningTime="2026-09-15 10:19:44.28658105 +0000 UTC m=+102.980736730"
Sep 15 10:19:44.743928 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:44.743894    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:44.744200 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:44.744177    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:19:45.067620 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:19:45.067544    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dszpz"
Sep 15 10:20:04.749208 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:04.749174    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:20:04.752998 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:04.752978    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5fc745c9c8-5v7b4"
Sep 15 10:20:26.485410 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:26.485373    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:26.503900 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:26.503876    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:27.447634 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:27.447608    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:40.290912 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.290874    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 10:20:40.291351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.291154    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="alertmanager" containerID="cri-o://8ffc96d412d3696c3cc28702df40523bba1bea358fdd50cfe244559038bce4df" gracePeriod=120
Sep 15 10:20:40.291351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.291227    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy-metric" containerID="cri-o://9a3d760d2a61cbb89bb7a30495ed9246f480791d11a7f73945399d5b916d42bd" gracePeriod=120
Sep 15 10:20:40.291351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.291279    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="prom-label-proxy" containerID="cri-o://72d664ff0f83aec8ae578dbdf0e639ba20a90a75bc1809f1e2fb68ab324aaeeb" gracePeriod=120
Sep 15 10:20:40.291351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.291297    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy-web" containerID="cri-o://ee04cfc9f56e3d37a7092f71413512636749651edacdb2acee78c2b236580585" gracePeriod=120
Sep 15 10:20:40.291351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.291313    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="config-reloader" containerID="cri-o://7ef7eca43c175c3cbf5f78763939c1942c6e9e038f8c5ad389becb6398dc5919" gracePeriod=120
Sep 15 10:20:40.291583 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.291376    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy" containerID="cri-o://c90243e4d3ee38feff3555cbc91027ca6e0b144a38e6b2008b5f4aac69bde78f" gracePeriod=120
Sep 15 10:20:40.488573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.488535    2584 generic.go:356] "Generic (PLEG): container finished" podID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerID="72d664ff0f83aec8ae578dbdf0e639ba20a90a75bc1809f1e2fb68ab324aaeeb" exitCode=0
Sep 15 10:20:40.488573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.488565    2584 generic.go:356] "Generic (PLEG): container finished" podID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerID="c90243e4d3ee38feff3555cbc91027ca6e0b144a38e6b2008b5f4aac69bde78f" exitCode=0
Sep 15 10:20:40.488573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.488574    2584 generic.go:356] "Generic (PLEG): container finished" podID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerID="7ef7eca43c175c3cbf5f78763939c1942c6e9e038f8c5ad389becb6398dc5919" exitCode=0
Sep 15 10:20:40.488573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.488582    2584 generic.go:356] "Generic (PLEG): container finished" podID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerID="8ffc96d412d3696c3cc28702df40523bba1bea358fdd50cfe244559038bce4df" exitCode=0
Sep 15 10:20:40.488857 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.488603    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerDied","Data":"72d664ff0f83aec8ae578dbdf0e639ba20a90a75bc1809f1e2fb68ab324aaeeb"}
Sep 15 10:20:40.488857 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.488692    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerDied","Data":"c90243e4d3ee38feff3555cbc91027ca6e0b144a38e6b2008b5f4aac69bde78f"}
Sep 15 10:20:40.488857 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.488728    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerDied","Data":"7ef7eca43c175c3cbf5f78763939c1942c6e9e038f8c5ad389becb6398dc5919"}
Sep 15 10:20:40.488857 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:40.488754    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerDied","Data":"8ffc96d412d3696c3cc28702df40523bba1bea358fdd50cfe244559038bce4df"}
Sep 15 10:20:41.497810 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.497768    2584 generic.go:356] "Generic (PLEG): container finished" podID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerID="9a3d760d2a61cbb89bb7a30495ed9246f480791d11a7f73945399d5b916d42bd" exitCode=0
Sep 15 10:20:41.497810 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.497790    2584 generic.go:356] "Generic (PLEG): container finished" podID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerID="ee04cfc9f56e3d37a7092f71413512636749651edacdb2acee78c2b236580585" exitCode=0
Sep 15 10:20:41.498300 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.497846    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerDied","Data":"9a3d760d2a61cbb89bb7a30495ed9246f480791d11a7f73945399d5b916d42bd"}
Sep 15 10:20:41.498300 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.497939    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerDied","Data":"ee04cfc9f56e3d37a7092f71413512636749651edacdb2acee78c2b236580585"}
Sep 15 10:20:41.537314 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.537290    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:20:41.662643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662555    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-trusted-ca-bundle\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.662643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662601    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.662643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662621    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-tls-assets\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.662643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662648    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-volume\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.663003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662688    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-web\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.663003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662715    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-out\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.663003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662738    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.663003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662765    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-cluster-tls-config\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.663003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662832    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-web-config\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.663003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662858    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-metrics-client-ca\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.663003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662883    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.663003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662914    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvz5s\" (UniqueName: \"kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-kube-api-access-bvz5s\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.663003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.662947    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-main-db\") pod \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\" (UID: \"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700\") "
Sep 15 10:20:41.663457 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.663004    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:20:41.663457 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.663195    2584 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.664063 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.664024    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:20:41.664445 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.664421    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:20:41.665481 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.665452    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:41.665758 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.665733    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:20:41.665882 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.665856    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-out" (OuterVolumeSpecName: "config-out") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:20:41.666305 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.666271    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:41.666381 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.666324    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:41.666673 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.666644    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-kube-api-access-bvz5s" (OuterVolumeSpecName: "kube-api-access-bvz5s") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "kube-api-access-bvz5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:20:41.667167 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.667144    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:41.667725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.667699    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-volume" (OuterVolumeSpecName: "config-volume") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:41.670289 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.670270    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:41.676190 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.676165    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-web-config" (OuterVolumeSpecName: "web-config") pod "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" (UID: "5118ee90-ec86-4e54-9dfb-d4c6d8ec2700"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:41.764339 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764310    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-metrics-client-ca\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.764339 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764334    2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-main-tls\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.764339 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764343    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvz5s\" (UniqueName: \"kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-kube-api-access-bvz5s\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.764558 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764351    2584 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-alertmanager-main-db\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.764558 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764361    2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.764558 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764370    2584 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-tls-assets\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.764558 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764377    2584 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.764558 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764386    2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.764558 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764394    2584 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-config-out\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.764558 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764403    2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.764558 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764411    2584 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-cluster-tls-config\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:41.764558 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:41.764436    2584 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700-web-config\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:42.504367 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:42.504341    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 10:20:42.504733 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:42.504337    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5118ee90-ec86-4e54-9dfb-d4c6d8ec2700","Type":"ContainerDied","Data":"805220e956565bb8431e87ccb5ff9cda9eaa31027c113a3f553af772bbc82a6a"}
Sep 15 10:20:42.504733 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:42.504495    2584 scope.go:117] "RemoveContainer" containerID="72d664ff0f83aec8ae578dbdf0e639ba20a90a75bc1809f1e2fb68ab324aaeeb"
Sep 15 10:20:42.512712 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:42.512687    2584 scope.go:117] "RemoveContainer" containerID="9a3d760d2a61cbb89bb7a30495ed9246f480791d11a7f73945399d5b916d42bd"
Sep 15 10:20:42.519231 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:42.519214    2584 scope.go:117] "RemoveContainer" containerID="c90243e4d3ee38feff3555cbc91027ca6e0b144a38e6b2008b5f4aac69bde78f"
Sep 15 10:20:42.525615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:42.525584    2584 scope.go:117] "RemoveContainer" containerID="ee04cfc9f56e3d37a7092f71413512636749651edacdb2acee78c2b236580585"
Sep 15 10:20:42.527123 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:42.527106    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 10:20:42.531301 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:42.531279    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 10:20:42.533291 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:42.533268    2584 scope.go:117] "RemoveContainer" containerID="7ef7eca43c175c3cbf5f78763939c1942c6e9e038f8c5ad389becb6398dc5919"
Sep 15 10:20:42.539776 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:42.539761    2584 scope.go:117] "RemoveContainer" containerID="8ffc96d412d3696c3cc28702df40523bba1bea358fdd50cfe244559038bce4df"
Sep 15 10:20:42.546373 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:42.546356    2584 scope.go:117] "RemoveContainer" containerID="598ad50015967e3444ed6b51615ad72fdbc3ce6819429cfffda4dc8e33920298"
Sep 15 10:20:43.783697 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:43.783665    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" path="/var/lib/kubelet/pods/5118ee90-ec86-4e54-9dfb-d4c6d8ec2700/volumes"
Sep 15 10:20:44.511170 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.511129    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 10:20:44.511554 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.511516    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="prometheus" containerID="cri-o://b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223" gracePeriod=600
Sep 15 10:20:44.511719 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.511543    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27" gracePeriod=600
Sep 15 10:20:44.511719 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.511570    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy-web" containerID="cri-o://5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8" gracePeriod=600
Sep 15 10:20:44.511719 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.511577    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="thanos-sidecar" containerID="cri-o://c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af" gracePeriod=600
Sep 15 10:20:44.511719 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.511584    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy" containerID="cri-o://38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6" gracePeriod=600
Sep 15 10:20:44.511991 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.511762    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="config-reloader" containerID="cri-o://2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf" gracePeriod=600
Sep 15 10:20:44.750410 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.750385    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:44.887642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.887556    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-thanos-prometheus-http-client-file\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.887642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.887596    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-db\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.887642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.887613    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-trusted-ca-bundle\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.887645    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-1\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.887782    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-kube-rbac-proxy\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.887855    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwswn\" (UniqueName: \"kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-kube-api-access-mwswn\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.887883    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-metrics-client-certs\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.887940    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-web-config\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.887972    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-tls\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.887996    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-kubelet-serving-ca-bundle\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.887994    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888032    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-0\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888038    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888065    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config-out\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888090    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888117    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-grpc-tls\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888149    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-2\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888195    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888221    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888252    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-metrics-client-ca\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888282    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-tls-assets\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888311    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-serving-certs-ca-bundle\") pod \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\" (UID: \"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1\") "
Sep 15 10:20:44.888935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888456    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:20:44.888935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888708    2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.888935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888726    2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.888935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888741    2584 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.889335 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.888984    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:20:44.889399 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.889355    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:20:44.890298 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.890268    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-kube-api-access-mwswn" (OuterVolumeSpecName: "kube-api-access-mwswn") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "kube-api-access-mwswn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:20:44.890434 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.890300    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:20:44.890434 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.890336    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:44.891258 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.890986    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:20:44.891258 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.890992    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:44.891258 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.891083    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:44.891591 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.891423    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:20:44.891591 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.891573    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config-out" (OuterVolumeSpecName: "config-out") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:20:44.892559 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.892318    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:44.892559 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.892485    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:44.892559 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.892549    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:44.893422 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.893396    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:44.893516 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.893412    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config" (OuterVolumeSpecName: "config") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:44.893704 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.893682    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:20:44.900405 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.900385    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-web-config" (OuterVolumeSpecName: "web-config") pod "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" (UID: "eeb81b57-aea5-4ccb-ab60-c34a9ca988e1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:20:44.989922 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.989892    2584 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.989922 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.989919    2584 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.989929    2584 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-metrics-client-ca\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.989941    2584 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-tls-assets\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.989950    2584 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.989959    2584 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.989969    2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-db\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.989977    2584 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-kube-rbac-proxy\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.989985    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwswn\" (UniqueName: \"kubernetes.io/projected/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-kube-api-access-mwswn\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.989993    2584 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-metrics-client-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.990003    2584 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-web-config\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.990011    2584 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.990019    2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.990027    2584 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-config-out\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.990035    2584 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.990043    2584 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-secret-grpc-tls\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:44.990102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:44.990052    2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:20:45.519911 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.519880    2584 generic.go:356] "Generic (PLEG): container finished" podID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerID="fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27" exitCode=0
Sep 15 10:20:45.519911 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.519904    2584 generic.go:356] "Generic (PLEG): container finished" podID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerID="38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6" exitCode=0
Sep 15 10:20:45.519911 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.519911    2584 generic.go:356] "Generic (PLEG): container finished" podID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerID="5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8" exitCode=0
Sep 15 10:20:45.519911 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.519916    2584 generic.go:356] "Generic (PLEG): container finished" podID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerID="c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af" exitCode=0
Sep 15 10:20:45.519911 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.519922    2584 generic.go:356] "Generic (PLEG): container finished" podID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerID="2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf" exitCode=0
Sep 15 10:20:45.520194 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.519927    2584 generic.go:356] "Generic (PLEG): container finished" podID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerID="b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223" exitCode=0
Sep 15 10:20:45.520194 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.519953    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.520194 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.519967    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerDied","Data":"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"}
Sep 15 10:20:45.520194 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.520040    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerDied","Data":"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"}
Sep 15 10:20:45.520194 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.520072    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerDied","Data":"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"}
Sep 15 10:20:45.520194 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.520101    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerDied","Data":"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"}
Sep 15 10:20:45.520194 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.520116    2584 scope.go:117] "RemoveContainer" containerID="fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"
Sep 15 10:20:45.520194 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.520141    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerDied","Data":"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"}
Sep 15 10:20:45.520194 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.520181    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerDied","Data":"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"}
Sep 15 10:20:45.520527 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.520220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeb81b57-aea5-4ccb-ab60-c34a9ca988e1","Type":"ContainerDied","Data":"f60d15fde03288603bd5b00342c1b95532cb667bd39fc6b3151b1452d6d8bd0a"}
Sep 15 10:20:45.538842 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.538825    2584 scope.go:117] "RemoveContainer" containerID="38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"
Sep 15 10:20:45.546482 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.546464    2584 scope.go:117] "RemoveContainer" containerID="5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"
Sep 15 10:20:45.551465 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.551445    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 10:20:45.555296 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.555139    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 10:20:45.555706 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.555490    2584 scope.go:117] "RemoveContainer" containerID="c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"
Sep 15 10:20:45.562268 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.562250    2584 scope.go:117] "RemoveContainer" containerID="2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"
Sep 15 10:20:45.568879 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.568863    2584 scope.go:117] "RemoveContainer" containerID="b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"
Sep 15 10:20:45.575297 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575274    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 10:20:45.575638 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575622    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy-web"
Sep 15 10:20:45.575688 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575641    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy-web"
Sep 15 10:20:45.575688 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575656    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="init-config-reloader"
Sep 15 10:20:45.575688 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575665    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="init-config-reloader"
Sep 15 10:20:45.575688 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575680    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575689    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575700    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575708    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575720    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="config-reloader"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575728    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="config-reloader"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575743    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="prom-label-proxy"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575751    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="prom-label-proxy"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575761    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="thanos-sidecar"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575769    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="thanos-sidecar"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575785    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy-web"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575794    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy-web"
Sep 15 10:20:45.575828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575828    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="init-config-reloader"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575836    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="init-config-reloader"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575845    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy-metric"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575853    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy-metric"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575866    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="alertmanager"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575874    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="alertmanager"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575883    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="prometheus"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575891    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="prometheus"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575901    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy-thanos"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575908    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy-thanos"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575919    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="config-reloader"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.575928    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="config-reloader"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576004    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy-metric"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576018    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="config-reloader"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576027    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="alertmanager"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576037    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="prometheus"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576047    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy-web"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576058    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="kube-rbac-proxy"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576066    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy-web"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576075    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576085    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="config-reloader"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576093    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="thanos-sidecar"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576101    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" containerName="kube-rbac-proxy-thanos"
Sep 15 10:20:45.576162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.576111    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5118ee90-ec86-4e54-9dfb-d4c6d8ec2700" containerName="prom-label-proxy"
Sep 15 10:20:45.577360 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.577344    2584 scope.go:117] "RemoveContainer" containerID="0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"
Sep 15 10:20:45.581534 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.581516    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.584179 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.584095    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 10:20:45.584179 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.584164    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 10:20:45.584405 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.584164    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 10:20:45.584405 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.584276    2584 scope.go:117] "RemoveContainer" containerID="fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"
Sep 15 10:20:45.584405 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.584322    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 10:20:45.584405 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.584365    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 10:20:45.584609 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.584526    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bd4g57ichahks\""
Sep 15 10:20:45.584609 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:20:45.584567    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": container with ID starting with fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27 not found: ID does not exist" containerID="fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"
Sep 15 10:20:45.584709 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.584603    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"} err="failed to get container status \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": rpc error: code = NotFound desc = could not find container \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": container with ID starting with fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27 not found: ID does not exist"
Sep 15 10:20:45.584709 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.584648    2584 scope.go:117] "RemoveContainer" containerID="38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"
Sep 15 10:20:45.584838 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.584813    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 10:20:45.585001 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:20:45.584964    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": container with ID starting with 38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6 not found: ID does not exist" containerID="38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"
Sep 15 10:20:45.585065 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585010    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"} err="failed to get container status \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": rpc error: code = NotFound desc = could not find container \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": container with ID starting with 38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6 not found: ID does not exist"
Sep 15 10:20:45.585065 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585032    2584 scope.go:117] "RemoveContainer" containerID="5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"
Sep 15 10:20:45.585146 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585127    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 10:20:45.585146 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585134    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m7zwv\""
Sep 15 10:20:45.585423 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:20:45.585393    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": container with ID starting with 5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8 not found: ID does not exist" containerID="5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"
Sep 15 10:20:45.585518 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585417    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"} err="failed to get container status \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": rpc error: code = NotFound desc = could not find container \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": container with ID starting with 5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8 not found: ID does not exist"
Sep 15 10:20:45.585518 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585438    2584 scope.go:117] "RemoveContainer" containerID="c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"
Sep 15 10:20:45.585639 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585550    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 10:20:45.585639 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585600    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 10:20:45.585639 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585604    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 10:20:45.585790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585670    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 10:20:45.585790 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:20:45.585733    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": container with ID starting with c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af not found: ID does not exist" containerID="c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"
Sep 15 10:20:45.585790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585550    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 10:20:45.585790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585758    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"} err="failed to get container status \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": rpc error: code = NotFound desc = could not find container \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": container with ID starting with c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af not found: ID does not exist"
Sep 15 10:20:45.585790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.585779    2584 scope.go:117] "RemoveContainer" containerID="2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"
Sep 15 10:20:45.586074 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:20:45.586055    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": container with ID starting with 2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf not found: ID does not exist" containerID="2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"
Sep 15 10:20:45.586116 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.586081    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"} err="failed to get container status \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": rpc error: code = NotFound desc = could not find container \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": container with ID starting with 2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf not found: ID does not exist"
Sep 15 10:20:45.586116 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.586095    2584 scope.go:117] "RemoveContainer" containerID="b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"
Sep 15 10:20:45.586360 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:20:45.586336    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": container with ID starting with b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223 not found: ID does not exist" containerID="b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"
Sep 15 10:20:45.586462 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.586362    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"} err="failed to get container status \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": rpc error: code = NotFound desc = could not find container \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": container with ID starting with b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223 not found: ID does not exist"
Sep 15 10:20:45.586462 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.586377    2584 scope.go:117] "RemoveContainer" containerID="0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"
Sep 15 10:20:45.586597 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:20:45.586578    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": container with ID starting with 0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19 not found: ID does not exist" containerID="0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"
Sep 15 10:20:45.586635 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.586601    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"} err="failed to get container status \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": rpc error: code = NotFound desc = could not find container \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": container with ID starting with 0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19 not found: ID does not exist"
Sep 15 10:20:45.586635 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.586614    2584 scope.go:117] "RemoveContainer" containerID="fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"
Sep 15 10:20:45.586927 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.586902    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"} err="failed to get container status \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": rpc error: code = NotFound desc = could not find container \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": container with ID starting with fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27 not found: ID does not exist"
Sep 15 10:20:45.587033 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.586929    2584 scope.go:117] "RemoveContainer" containerID="38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"
Sep 15 10:20:45.587271 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.587252    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"} err="failed to get container status \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": rpc error: code = NotFound desc = could not find container \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": container with ID starting with 38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6 not found: ID does not exist"
Sep 15 10:20:45.587356 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.587273    2584 scope.go:117] "RemoveContainer" containerID="5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"
Sep 15 10:20:45.587547 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.587514    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"} err="failed to get container status \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": rpc error: code = NotFound desc = could not find container \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": container with ID starting with 5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8 not found: ID does not exist"
Sep 15 10:20:45.587547 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.587546    2584 scope.go:117] "RemoveContainer" containerID="c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"
Sep 15 10:20:45.587961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.587921    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"} err="failed to get container status \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": rpc error: code = NotFound desc = could not find container \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": container with ID starting with c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af not found: ID does not exist"
Sep 15 10:20:45.587961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.587948    2584 scope.go:117] "RemoveContainer" containerID="2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"
Sep 15 10:20:45.588243 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.588214    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"} err="failed to get container status \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": rpc error: code = NotFound desc = could not find container \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": container with ID starting with 2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf not found: ID does not exist"
Sep 15 10:20:45.588329 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.588246    2584 scope.go:117] "RemoveContainer" containerID="b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"
Sep 15 10:20:45.588521 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.588501    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 10:20:45.588624 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.588572    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"} err="failed to get container status \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": rpc error: code = NotFound desc = could not find container \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": container with ID starting with b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223 not found: ID does not exist"
Sep 15 10:20:45.588624 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.588595    2584 scope.go:117] "RemoveContainer" containerID="0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"
Sep 15 10:20:45.588862 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.588836    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"} err="failed to get container status \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": rpc error: code = NotFound desc = could not find container \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": container with ID starting with 0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19 not found: ID does not exist"
Sep 15 10:20:45.588862 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.588862    2584 scope.go:117] "RemoveContainer" containerID="fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"
Sep 15 10:20:45.589228 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.589201    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"} err="failed to get container status \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": rpc error: code = NotFound desc = could not find container \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": container with ID starting with fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27 not found: ID does not exist"
Sep 15 10:20:45.589326 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.589229    2584 scope.go:117] "RemoveContainer" containerID="38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"
Sep 15 10:20:45.589535 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.589515    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"} err="failed to get container status \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": rpc error: code = NotFound desc = could not find container \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": container with ID starting with 38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6 not found: ID does not exist"
Sep 15 10:20:45.589624 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.589537    2584 scope.go:117] "RemoveContainer" containerID="5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"
Sep 15 10:20:45.589827 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.589788    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"} err="failed to get container status \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": rpc error: code = NotFound desc = could not find container \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": container with ID starting with 5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8 not found: ID does not exist"
Sep 15 10:20:45.589827 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.589827    2584 scope.go:117] "RemoveContainer" containerID="c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"
Sep 15 10:20:45.590098 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.590075    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"} err="failed to get container status \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": rpc error: code = NotFound desc = could not find container \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": container with ID starting with c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af not found: ID does not exist"
Sep 15 10:20:45.590161 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.590100    2584 scope.go:117] "RemoveContainer" containerID="2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"
Sep 15 10:20:45.590161 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.590108    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 10:20:45.590373 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.590348    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"} err="failed to get container status \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": rpc error: code = NotFound desc = could not find container \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": container with ID starting with 2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf not found: ID does not exist"
Sep 15 10:20:45.590373 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.590369    2584 scope.go:117] "RemoveContainer" containerID="b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"
Sep 15 10:20:45.590757 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.590684    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"} err="failed to get container status \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": rpc error: code = NotFound desc = could not find container \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": container with ID starting with b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223 not found: ID does not exist"
Sep 15 10:20:45.590757 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.590744    2584 scope.go:117] "RemoveContainer" containerID="0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"
Sep 15 10:20:45.591125 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.591096    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"} err="failed to get container status \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": rpc error: code = NotFound desc = could not find container \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": container with ID starting with 0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19 not found: ID does not exist"
Sep 15 10:20:45.591218 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.591126    2584 scope.go:117] "RemoveContainer" containerID="fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"
Sep 15 10:20:45.591408 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.591381    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"} err="failed to get container status \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": rpc error: code = NotFound desc = could not find container \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": container with ID starting with fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27 not found: ID does not exist"
Sep 15 10:20:45.591480 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.591409    2584 scope.go:117] "RemoveContainer" containerID="38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"
Sep 15 10:20:45.591716 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.591693    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"} err="failed to get container status \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": rpc error: code = NotFound desc = could not find container \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": container with ID starting with 38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6 not found: ID does not exist"
Sep 15 10:20:45.591900 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.591716    2584 scope.go:117] "RemoveContainer" containerID="5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"
Sep 15 10:20:45.592069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.592041    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"} err="failed to get container status \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": rpc error: code = NotFound desc = could not find container \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": container with ID starting with 5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8 not found: ID does not exist"
Sep 15 10:20:45.592069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.592069    2584 scope.go:117] "RemoveContainer" containerID="c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"
Sep 15 10:20:45.592466 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.592368    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"} err="failed to get container status \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": rpc error: code = NotFound desc = could not find container \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": container with ID starting with c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af not found: ID does not exist"
Sep 15 10:20:45.592466 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.592394    2584 scope.go:117] "RemoveContainer" containerID="2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"
Sep 15 10:20:45.592822 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.592785    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"} err="failed to get container status \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": rpc error: code = NotFound desc = could not find container \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": container with ID starting with 2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf not found: ID does not exist"
Sep 15 10:20:45.592899 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.592824    2584 scope.go:117] "RemoveContainer" containerID="b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"
Sep 15 10:20:45.593074 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.593057    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"} err="failed to get container status \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": rpc error: code = NotFound desc = could not find container \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": container with ID starting with b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223 not found: ID does not exist"
Sep 15 10:20:45.593114 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.593076    2584 scope.go:117] "RemoveContainer" containerID="0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"
Sep 15 10:20:45.593312 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.593294    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"} err="failed to get container status \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": rpc error: code = NotFound desc = could not find container \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": container with ID starting with 0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19 not found: ID does not exist"
Sep 15 10:20:45.593364 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.593308    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 10:20:45.593398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.593315    2584 scope.go:117] "RemoveContainer" containerID="fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"
Sep 15 10:20:45.593624 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.593608    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"} err="failed to get container status \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": rpc error: code = NotFound desc = could not find container \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": container with ID starting with fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27 not found: ID does not exist"
Sep 15 10:20:45.593624 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.593623    2584 scope.go:117] "RemoveContainer" containerID="38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"
Sep 15 10:20:45.593863 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.593844    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"} err="failed to get container status \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": rpc error: code = NotFound desc = could not find container \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": container with ID starting with 38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6 not found: ID does not exist"
Sep 15 10:20:45.593927 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.593864    2584 scope.go:117] "RemoveContainer" containerID="5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"
Sep 15 10:20:45.594092 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.594074    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"} err="failed to get container status \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": rpc error: code = NotFound desc = could not find container \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": container with ID starting with 5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8 not found: ID does not exist"
Sep 15 10:20:45.594140 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.594092    2584 scope.go:117] "RemoveContainer" containerID="c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"
Sep 15 10:20:45.594347 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.594329    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"} err="failed to get container status \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": rpc error: code = NotFound desc = could not find container \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": container with ID starting with c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af not found: ID does not exist"
Sep 15 10:20:45.594400 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.594347    2584 scope.go:117] "RemoveContainer" containerID="2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"
Sep 15 10:20:45.594562 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.594548    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"} err="failed to get container status \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": rpc error: code = NotFound desc = could not find container \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": container with ID starting with 2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf not found: ID does not exist"
Sep 15 10:20:45.594605 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.594562    2584 scope.go:117] "RemoveContainer" containerID="b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"
Sep 15 10:20:45.594811 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.594782    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"} err="failed to get container status \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": rpc error: code = NotFound desc = could not find container \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": container with ID starting with b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223 not found: ID does not exist"
Sep 15 10:20:45.594850 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.594821    2584 scope.go:117] "RemoveContainer" containerID="0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"
Sep 15 10:20:45.595049 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.595031    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"} err="failed to get container status \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": rpc error: code = NotFound desc = could not find container \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": container with ID starting with 0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19 not found: ID does not exist"
Sep 15 10:20:45.595049 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.595048    2584 scope.go:117] "RemoveContainer" containerID="fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"
Sep 15 10:20:45.595327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.595304    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27"} err="failed to get container status \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": rpc error: code = NotFound desc = could not find container \"fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27\": container with ID starting with fd9d249b344ed7a4bd2e5cbd744972c0600a162c8c0fe93d1790be6911ac9a27 not found: ID does not exist"
Sep 15 10:20:45.595327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.595327    2584 scope.go:117] "RemoveContainer" containerID="38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"
Sep 15 10:20:45.595617 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.595596    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6"} err="failed to get container status \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": rpc error: code = NotFound desc = could not find container \"38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6\": container with ID starting with 38b3a7d24ea597274cc1e6bc3a9aeae9f040da9c4e6d3051a15f3360b5c18aa6 not found: ID does not exist"
Sep 15 10:20:45.595617 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.595617    2584 scope.go:117] "RemoveContainer" containerID="5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"
Sep 15 10:20:45.595870 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.595846    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8"} err="failed to get container status \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": rpc error: code = NotFound desc = could not find container \"5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8\": container with ID starting with 5784f621219d1d2739f7ce0d3c6d804b00c7eede05f9a5405e055802e82c70f8 not found: ID does not exist"
Sep 15 10:20:45.595928 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.595874    2584 scope.go:117] "RemoveContainer" containerID="c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"
Sep 15 10:20:45.596091 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.596075    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af"} err="failed to get container status \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": rpc error: code = NotFound desc = could not find container \"c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af\": container with ID starting with c047241e13c8ac4118fd12bcc3e201b0833ff22d71a661546a06c7f2b0b928af not found: ID does not exist"
Sep 15 10:20:45.596147 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.596090    2584 scope.go:117] "RemoveContainer" containerID="2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"
Sep 15 10:20:45.596342 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.596321    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf"} err="failed to get container status \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": rpc error: code = NotFound desc = could not find container \"2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf\": container with ID starting with 2be0a7243b52ad3d97405aaf99d62c4265efd0757d14f7a75af73c8526d12ddf not found: ID does not exist"
Sep 15 10:20:45.596342 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.596341    2584 scope.go:117] "RemoveContainer" containerID="b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"
Sep 15 10:20:45.596613 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.596589    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223"} err="failed to get container status \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": rpc error: code = NotFound desc = could not find container \"b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223\": container with ID starting with b36b35b09ee62fc473946c642d6e6ed82f47256a19b69745097040b7f5807223 not found: ID does not exist"
Sep 15 10:20:45.596665 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.596615    2584 scope.go:117] "RemoveContainer" containerID="0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"
Sep 15 10:20:45.596953 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.596925    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19"} err="failed to get container status \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": rpc error: code = NotFound desc = could not find container \"0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19\": container with ID starting with 0bea5b74b1f15ef27f664530c933d6aafcc371af9c1d672bf1250f6a5f3a6a19 not found: ID does not exist"
Sep 15 10:20:45.695057 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695023    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695057 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695056    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695246 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695084    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695246 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695100    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695246 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695140    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695246 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695160    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695246 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695178    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695246 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695204    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695246 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695246 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695238    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695310    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-config\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695339    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695359    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695387    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695410    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695424    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx89k\" (UniqueName: \"kubernetes.io/projected/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-kube-api-access-rx89k\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695705 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695499    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.695705 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.695515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.788180 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.788098    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeb81b57-aea5-4ccb-ab60-c34a9ca988e1" path="/var/lib/kubelet/pods/eeb81b57-aea5-4ccb-ab60-c34a9ca988e1/volumes"
Sep 15 10:20:45.795903 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.795873    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-config\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.796032 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.795914    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.796032 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.795936    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.796117 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796028    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.796117 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796087    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.796262 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796244    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx89k\" (UniqueName: \"kubernetes.io/projected/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-kube-api-access-rx89k\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796283    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796366    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796403    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796426    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796466    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796490    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796518    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796545    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796568    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796593    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796615    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796618    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796614    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.796905    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797934 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.797328    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797934 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.797371    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.797934 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.797560    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.799624 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.799581    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.799839 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.799789    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.799937 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.799795    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.801304 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.801280    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.801439 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.801297    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.801571 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.801360    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-config\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.801682 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.801422    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.801777 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.801607    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.801863 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.801721    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.801863 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.801817    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.801863 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.801849    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.801959 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.801866    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.802323 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.802302    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.802968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.802946    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.803045 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.802984    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.804703 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.804682    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx89k\" (UniqueName: \"kubernetes.io/projected/7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1-kube-api-access-rx89k\") pod \"prometheus-k8s-0\" (UID: \"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:45.893422 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:45.893384    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:20:46.173244 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:46.173147    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 10:20:46.175301 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:20:46.175272    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c53a75d_78ad_41af_8d4a_3b4f03fe0ec1.slice/crio-9156521d64271fc667190f54dae5dcbd928f2706ad24751bffd4e921e20019bb WatchSource:0}: Error finding container 9156521d64271fc667190f54dae5dcbd928f2706ad24751bffd4e921e20019bb: Status 404 returned error can't find the container with id 9156521d64271fc667190f54dae5dcbd928f2706ad24751bffd4e921e20019bb
Sep 15 10:20:46.525604 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:46.525566    2584 generic.go:356] "Generic (PLEG): container finished" podID="7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1" containerID="914189b6865cd7bf8f1de238008748c684cf01b6c516cb1fb97694f7b293f418" exitCode=0
Sep 15 10:20:46.525762 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:46.525650    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1","Type":"ContainerDied","Data":"914189b6865cd7bf8f1de238008748c684cf01b6c516cb1fb97694f7b293f418"}
Sep 15 10:20:46.525762 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:46.525735    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1","Type":"ContainerStarted","Data":"9156521d64271fc667190f54dae5dcbd928f2706ad24751bffd4e921e20019bb"}
Sep 15 10:20:47.534225 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:47.534190    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1","Type":"ContainerStarted","Data":"6cc4ba94761acf17d93569e69bff9396b4e8044ade85e6c797d90f05680be01b"}
Sep 15 10:20:47.534643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:47.534259    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1","Type":"ContainerStarted","Data":"005326db46b3c94658f32bf17777aeeffa0c1c3cbd483f1e1a2f1567b9c32dba"}
Sep 15 10:20:47.534643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:47.534287    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1","Type":"ContainerStarted","Data":"aa45fe26090417c371b5d87017bded75494dc97d496b352fd0f07a4afe15df2d"}
Sep 15 10:20:47.534643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:47.534314    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1","Type":"ContainerStarted","Data":"78d65dfde3bf666a1dcbe9d7ce26864bbc6aac5b5d5d945a7d933dc53dede78d"}
Sep 15 10:20:47.534643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:47.534341    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1","Type":"ContainerStarted","Data":"54e55d6ac8500dec9487ac44a88d39efba22f239d4b0b7b657077047241d859e"}
Sep 15 10:20:47.534643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:47.534364    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1","Type":"ContainerStarted","Data":"1553e70f7a3a7324b74c9c55268090ff018f7c711fbd184f74bbd233764d7ab0"}
Sep 15 10:20:47.563462 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:47.563414    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.545773373 podStartE2EDuration="2.563397741s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.624368ms" isStatefulPod=false podCreationTimestamp="2026-09-15 10:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:20:47.561397359 +0000 UTC m=+166.255553041" watchObservedRunningTime="2026-09-15 10:20:47.563397741 +0000 UTC m=+166.257553422"
Sep 15 10:20:50.894126 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:20:50.894074    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:21:45.894302 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:21:45.894268    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:21:45.909355 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:21:45.909329    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:21:46.760995 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:21:46.760967    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 10:23:01.655105 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:01.655074    2584 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 10:23:08.472217 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.472181    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"]
Sep 15 10:23:08.475619 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.475588    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.478823 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.478771    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 10:23:08.478823 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.478814    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 10:23:08.479000 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.478834    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 10:23:08.479000 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.478814    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 10:23:08.479000 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.478870    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cdltk\""
Sep 15 10:23:08.479157 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.479036    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 10:23:08.483283 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.483264    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"]
Sep 15 10:23:08.611451 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.611411    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7lpw\" (UniqueName: \"kubernetes.io/projected/f4eb586c-020c-48ac-9040-9eec68995175-kube-api-access-s7lpw\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.611635 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.611459    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4eb586c-020c-48ac-9040-9eec68995175-metrics-cert\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.611635 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.611489    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4eb586c-020c-48ac-9040-9eec68995175-cert\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.611635 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.611511    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4eb586c-020c-48ac-9040-9eec68995175-manager-config\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.712946 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.712913    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7lpw\" (UniqueName: \"kubernetes.io/projected/f4eb586c-020c-48ac-9040-9eec68995175-kube-api-access-s7lpw\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.713135 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.712956    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4eb586c-020c-48ac-9040-9eec68995175-metrics-cert\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.713135 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.712976    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4eb586c-020c-48ac-9040-9eec68995175-cert\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.713135 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.712998    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4eb586c-020c-48ac-9040-9eec68995175-manager-config\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.713725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.713701    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4eb586c-020c-48ac-9040-9eec68995175-manager-config\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.715396 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.715373    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4eb586c-020c-48ac-9040-9eec68995175-metrics-cert\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.715575 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.715551    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4eb586c-020c-48ac-9040-9eec68995175-cert\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.720620 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.720598    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7lpw\" (UniqueName: \"kubernetes.io/projected/f4eb586c-020c-48ac-9040-9eec68995175-kube-api-access-s7lpw\") pod \"lws-controller-manager-85d6978c87-4hl5x\" (UID: \"f4eb586c-020c-48ac-9040-9eec68995175\") " pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:08.785776 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:08.785734    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:09.063544 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:09.063470    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"]
Sep 15 10:23:09.066329 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:23:09.066298    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4eb586c_020c_48ac_9040_9eec68995175.slice/crio-96a5a4007b4d599517466bfa49e425c5eef9cea8a81bacc990d32a41673d97e3 WatchSource:0}: Error finding container 96a5a4007b4d599517466bfa49e425c5eef9cea8a81bacc990d32a41673d97e3: Status 404 returned error can't find the container with id 96a5a4007b4d599517466bfa49e425c5eef9cea8a81bacc990d32a41673d97e3
Sep 15 10:23:09.068425 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:09.068407    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 10:23:10.036272 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:10.036234    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x" event={"ID":"f4eb586c-020c-48ac-9040-9eec68995175","Type":"ContainerStarted","Data":"96a5a4007b4d599517466bfa49e425c5eef9cea8a81bacc990d32a41673d97e3"}
Sep 15 10:23:12.046068 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.046029    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x" event={"ID":"f4eb586c-020c-48ac-9040-9eec68995175","Type":"ContainerStarted","Data":"5ae96d5c7a5a7f605170c9ee60d0863d438b381085bfd3a9d96a0b5e78d5be19"}
Sep 15 10:23:12.046550 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.046159    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:12.063187 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.063137    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x" podStartSLOduration=2.019054427 podStartE2EDuration="4.06312351s" totalImagesPullingTime="2.044069083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:23:12.061841205 +0000 UTC m=+310.755996885" watchObservedRunningTime="2026-09-15 10:23:12.06312351 +0000 UTC m=+310.757279267"
Sep 15 10:23:12.267560 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.267527    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"]
Sep 15 10:23:12.270680 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.270657    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.272901 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.272882    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 10:23:12.273083 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.273063    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 10:23:12.273199 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.273183    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jkrwv\""
Sep 15 10:23:12.273373 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.273355    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 10:23:12.273449 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.273375    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 10:23:12.287583 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.287560    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"]
Sep 15 10:23:12.445917 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.445826    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-manifests\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.445917 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.445873    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-tmp\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.445917 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.445891    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whgfr\" (UniqueName: \"kubernetes.io/projected/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-kube-api-access-whgfr\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.446130 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.446018    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.446130 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.446050    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.547192 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.547153    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-manifests\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.547192 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.547194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-tmp\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.547445 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.547213    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whgfr\" (UniqueName: \"kubernetes.io/projected/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-kube-api-access-whgfr\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.547445 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.547268    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.547445 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.547285    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.547587 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.547554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-manifests\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.549624 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.549597    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-tmp\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.549738 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.549688    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.549738 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.549719    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.558112 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.558088    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whgfr\" (UniqueName: \"kubernetes.io/projected/0556b2fa-e5ad-4e59-b859-7c3a133d93cf-kube-api-access-whgfr\") pod \"opendatahub-operator-controller-manager-6699d7bd9c-46dfj\" (UID: \"0556b2fa-e5ad-4e59-b859-7c3a133d93cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.581213 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.581193    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:12.862120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:12.862088    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"]
Sep 15 10:23:12.864523 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:23:12.864499    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0556b2fa_e5ad_4e59_b859_7c3a133d93cf.slice/crio-df82df92dfe5d7a13ba5e22efaf3df46a3f1bc57585d831b05f695fa5dbb7b0d WatchSource:0}: Error finding container df82df92dfe5d7a13ba5e22efaf3df46a3f1bc57585d831b05f695fa5dbb7b0d: Status 404 returned error can't find the container with id df82df92dfe5d7a13ba5e22efaf3df46a3f1bc57585d831b05f695fa5dbb7b0d
Sep 15 10:23:13.051563 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:13.051528    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj" event={"ID":"0556b2fa-e5ad-4e59-b859-7c3a133d93cf","Type":"ContainerStarted","Data":"df82df92dfe5d7a13ba5e22efaf3df46a3f1bc57585d831b05f695fa5dbb7b0d"}
Sep 15 10:23:16.065342 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:16.065307    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj" event={"ID":"0556b2fa-e5ad-4e59-b859-7c3a133d93cf","Type":"ContainerStarted","Data":"7bf71ae1b36e3c048b0198c76269ca745d065533185da15daa007c81870883bc"}
Sep 15 10:23:17.070717 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:17.070678    2584 generic.go:356] "Generic (PLEG): container finished" podID="0556b2fa-e5ad-4e59-b859-7c3a133d93cf" containerID="7bf71ae1b36e3c048b0198c76269ca745d065533185da15daa007c81870883bc" exitCode=0
Sep 15 10:23:17.071121 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:17.070768    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj" event={"ID":"0556b2fa-e5ad-4e59-b859-7c3a133d93cf","Type":"ContainerDied","Data":"7bf71ae1b36e3c048b0198c76269ca745d065533185da15daa007c81870883bc"}
Sep 15 10:23:18.076703 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:18.076663    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj" event={"ID":"0556b2fa-e5ad-4e59-b859-7c3a133d93cf","Type":"ContainerStarted","Data":"356d5e23ed8c3e00be96dbfb0ba21b181c5c7ddac0b825822bc92aa10a98ce87"}
Sep 15 10:23:18.077136 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:18.076790    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:18.098666 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:18.098610    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj" podStartSLOduration=1.681878939 podStartE2EDuration="6.098588683s" totalImagesPullingTime="3.374409689s" totalInitContainerRuntime="1.042300055s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:23:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:23:18.096624702 +0000 UTC m=+316.790780387" watchObservedRunningTime="2026-09-15 10:23:18.098588683 +0000 UTC m=+316.792744362"
Sep 15 10:23:23.053660 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:23.053625    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85d6978c87-4hl5x"
Sep 15 10:23:29.082470 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:29.082434    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6699d7bd9c-46dfj"
Sep 15 10:23:31.656981 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.656947    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"]
Sep 15 10:23:31.745206 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.745175    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"]
Sep 15 10:23:31.745379 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.745308    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:31.747569 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.747543    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 10:23:31.747961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.747940    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hjdq9\""
Sep 15 10:23:31.748092 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.748029    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 10:23:31.748663 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.748638    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 10:23:31.748781 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.748637    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 10:23:31.810885 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.810859    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1e3360c-f192-4adf-a2f6-815658a66fc5-tmp\") pod \"kube-auth-proxy-58df565b49-s7f5p\" (UID: \"f1e3360c-f192-4adf-a2f6-815658a66fc5\") " pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:31.811035 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.810909    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ckl7\" (UniqueName: \"kubernetes.io/projected/f1e3360c-f192-4adf-a2f6-815658a66fc5-kube-api-access-7ckl7\") pod \"kube-auth-proxy-58df565b49-s7f5p\" (UID: \"f1e3360c-f192-4adf-a2f6-815658a66fc5\") " pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:31.811035 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.810945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e3360c-f192-4adf-a2f6-815658a66fc5-tls-certs\") pod \"kube-auth-proxy-58df565b49-s7f5p\" (UID: \"f1e3360c-f192-4adf-a2f6-815658a66fc5\") " pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:31.911524 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.911447    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ckl7\" (UniqueName: \"kubernetes.io/projected/f1e3360c-f192-4adf-a2f6-815658a66fc5-kube-api-access-7ckl7\") pod \"kube-auth-proxy-58df565b49-s7f5p\" (UID: \"f1e3360c-f192-4adf-a2f6-815658a66fc5\") " pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:31.911524 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.911492    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e3360c-f192-4adf-a2f6-815658a66fc5-tls-certs\") pod \"kube-auth-proxy-58df565b49-s7f5p\" (UID: \"f1e3360c-f192-4adf-a2f6-815658a66fc5\") " pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:31.911712 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.911545    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1e3360c-f192-4adf-a2f6-815658a66fc5-tmp\") pod \"kube-auth-proxy-58df565b49-s7f5p\" (UID: \"f1e3360c-f192-4adf-a2f6-815658a66fc5\") " pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:31.911712 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:23:31.911663    2584 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 15 10:23:31.911779 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:23:31.911738    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e3360c-f192-4adf-a2f6-815658a66fc5-tls-certs podName:f1e3360c-f192-4adf-a2f6-815658a66fc5 nodeName:}" failed. No retries permitted until 2026-09-15 10:23:32.411717771 +0000 UTC m=+331.105873435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f1e3360c-f192-4adf-a2f6-815658a66fc5-tls-certs") pod "kube-auth-proxy-58df565b49-s7f5p" (UID: "f1e3360c-f192-4adf-a2f6-815658a66fc5") : secret "kube-auth-proxy-tls" not found
Sep 15 10:23:31.913751 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.913721    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1e3360c-f192-4adf-a2f6-815658a66fc5-tmp\") pod \"kube-auth-proxy-58df565b49-s7f5p\" (UID: \"f1e3360c-f192-4adf-a2f6-815658a66fc5\") " pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:31.921965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:31.921941    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ckl7\" (UniqueName: \"kubernetes.io/projected/f1e3360c-f192-4adf-a2f6-815658a66fc5-kube-api-access-7ckl7\") pod \"kube-auth-proxy-58df565b49-s7f5p\" (UID: \"f1e3360c-f192-4adf-a2f6-815658a66fc5\") " pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:32.416027 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:32.415992    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e3360c-f192-4adf-a2f6-815658a66fc5-tls-certs\") pod \"kube-auth-proxy-58df565b49-s7f5p\" (UID: \"f1e3360c-f192-4adf-a2f6-815658a66fc5\") " pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:32.418334 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:32.418304    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e3360c-f192-4adf-a2f6-815658a66fc5-tls-certs\") pod \"kube-auth-proxy-58df565b49-s7f5p\" (UID: \"f1e3360c-f192-4adf-a2f6-815658a66fc5\") " pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:32.654658 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:32.654622    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"
Sep 15 10:23:32.924340 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:32.924309    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58df565b49-s7f5p"]
Sep 15 10:23:32.927867 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:23:32.927842    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e3360c_f192_4adf_a2f6_815658a66fc5.slice/crio-1c89ed621f8eb12f6de610e85a248ae286af8dd78b6fbbffe4561c119dd7e785 WatchSource:0}: Error finding container 1c89ed621f8eb12f6de610e85a248ae286af8dd78b6fbbffe4561c119dd7e785: Status 404 returned error can't find the container with id 1c89ed621f8eb12f6de610e85a248ae286af8dd78b6fbbffe4561c119dd7e785
Sep 15 10:23:33.131767 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:33.131731    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p" event={"ID":"f1e3360c-f192-4adf-a2f6-815658a66fc5","Type":"ContainerStarted","Data":"1c89ed621f8eb12f6de610e85a248ae286af8dd78b6fbbffe4561c119dd7e785"}
Sep 15 10:23:36.147862 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:36.147821    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p" event={"ID":"f1e3360c-f192-4adf-a2f6-815658a66fc5","Type":"ContainerStarted","Data":"a6fa5d447cd0e3ec33e250730f4c39fc8b2b07633d66f4c6cdffc73d329c40fc"}
Sep 15 10:23:36.163449 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:36.163399    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58df565b49-s7f5p" podStartSLOduration=2.607324088 podStartE2EDuration="5.163383285s" totalImagesPullingTime="2.556059197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:23:36.162121607 +0000 UTC m=+334.856277297" watchObservedRunningTime="2026-09-15 10:23:36.163383285 +0000 UTC m=+334.857538964"
Sep 15 10:23:51.542761 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.542722    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"]
Sep 15 10:23:51.547744 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.547723    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"
Sep 15 10:23:51.550553 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.550516    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 15 10:23:51.550656 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.550556    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wcqbm\""
Sep 15 10:23:51.561102 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.561076    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"]
Sep 15 10:23:51.682164 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.682130    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/128887f8-37c7-4036-ac64-5644ffce6f38-cert\") pod \"kserve-controller-manager-d6f4b65b8-z6prd\" (UID: \"128887f8-37c7-4036-ac64-5644ffce6f38\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"
Sep 15 10:23:51.682332 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.682181    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgvp2\" (UniqueName: \"kubernetes.io/projected/128887f8-37c7-4036-ac64-5644ffce6f38-kube-api-access-wgvp2\") pod \"kserve-controller-manager-d6f4b65b8-z6prd\" (UID: \"128887f8-37c7-4036-ac64-5644ffce6f38\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"
Sep 15 10:23:51.782655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.782624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/128887f8-37c7-4036-ac64-5644ffce6f38-cert\") pod \"kserve-controller-manager-d6f4b65b8-z6prd\" (UID: \"128887f8-37c7-4036-ac64-5644ffce6f38\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"
Sep 15 10:23:51.782864 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.782701    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgvp2\" (UniqueName: \"kubernetes.io/projected/128887f8-37c7-4036-ac64-5644ffce6f38-kube-api-access-wgvp2\") pod \"kserve-controller-manager-d6f4b65b8-z6prd\" (UID: \"128887f8-37c7-4036-ac64-5644ffce6f38\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"
Sep 15 10:23:51.785420 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.785394    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/128887f8-37c7-4036-ac64-5644ffce6f38-cert\") pod \"kserve-controller-manager-d6f4b65b8-z6prd\" (UID: \"128887f8-37c7-4036-ac64-5644ffce6f38\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"
Sep 15 10:23:51.796185 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.796131    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgvp2\" (UniqueName: \"kubernetes.io/projected/128887f8-37c7-4036-ac64-5644ffce6f38-kube-api-access-wgvp2\") pod \"kserve-controller-manager-d6f4b65b8-z6prd\" (UID: \"128887f8-37c7-4036-ac64-5644ffce6f38\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"
Sep 15 10:23:51.858099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:51.858049    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"
Sep 15 10:23:52.133039 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:52.132967    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"]
Sep 15 10:23:52.135956 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:23:52.135927    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128887f8_37c7_4036_ac64_5644ffce6f38.slice/crio-bb5ed5dc4667a025c3025cb7a1b8fb4b87ce7e1ef21d76b63a1d1d618bced2f3 WatchSource:0}: Error finding container bb5ed5dc4667a025c3025cb7a1b8fb4b87ce7e1ef21d76b63a1d1d618bced2f3: Status 404 returned error can't find the container with id bb5ed5dc4667a025c3025cb7a1b8fb4b87ce7e1ef21d76b63a1d1d618bced2f3
Sep 15 10:23:52.220576 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:52.220539    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd" event={"ID":"128887f8-37c7-4036-ac64-5644ffce6f38","Type":"ContainerStarted","Data":"bb5ed5dc4667a025c3025cb7a1b8fb4b87ce7e1ef21d76b63a1d1d618bced2f3"}
Sep 15 10:23:54.231320 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:54.231285    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd" event={"ID":"128887f8-37c7-4036-ac64-5644ffce6f38","Type":"ContainerStarted","Data":"fd9121c1aeb206af1487db5694281083c963bd52895a5ffe4aa5f238d0a80667"}
Sep 15 10:23:54.231780 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:54.231424    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"
Sep 15 10:23:54.248729 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:23:54.248684    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd" podStartSLOduration=1.247373287 podStartE2EDuration="3.248668794s" totalImagesPullingTime="2.001295507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:23:54.247349436 +0000 UTC m=+352.941505116" watchObservedRunningTime="2026-09-15 10:23:54.248668794 +0000 UTC m=+352.942824474"
Sep 15 10:24:25.240230 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:24:25.240197    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6prd"
Sep 15 10:25:14.920052 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:14.920019    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"]
Sep 15 10:25:14.923026 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:14.923008    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:14.925675 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:14.925650    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 10:25:14.925781 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:14.925677    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 10:25:14.926854 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:14.926833    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8c94h\""
Sep 15 10:25:14.926854 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:14.926849    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 10:25:14.926988 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:14.926866    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 10:25:14.929332 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:14.929313    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"]
Sep 15 10:25:15.046782 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:15.046737    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b677254b-e134-46bb-9ea4-a6c2faeaa047-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dhj2d\" (UID: \"b677254b-e134-46bb-9ea4-a6c2faeaa047\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:15.046782 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:15.046783    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zshst\" (UniqueName: \"kubernetes.io/projected/b677254b-e134-46bb-9ea4-a6c2faeaa047-kube-api-access-zshst\") pod \"kuadrant-console-plugin-fbdd4f445-dhj2d\" (UID: \"b677254b-e134-46bb-9ea4-a6c2faeaa047\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:15.047018 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:15.046906    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b677254b-e134-46bb-9ea4-a6c2faeaa047-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dhj2d\" (UID: \"b677254b-e134-46bb-9ea4-a6c2faeaa047\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:15.148432 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:15.148392    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b677254b-e134-46bb-9ea4-a6c2faeaa047-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dhj2d\" (UID: \"b677254b-e134-46bb-9ea4-a6c2faeaa047\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:15.148624 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:15.148474    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b677254b-e134-46bb-9ea4-a6c2faeaa047-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dhj2d\" (UID: \"b677254b-e134-46bb-9ea4-a6c2faeaa047\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:15.148624 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:15.148491    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshst\" (UniqueName: \"kubernetes.io/projected/b677254b-e134-46bb-9ea4-a6c2faeaa047-kube-api-access-zshst\") pod \"kuadrant-console-plugin-fbdd4f445-dhj2d\" (UID: \"b677254b-e134-46bb-9ea4-a6c2faeaa047\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:15.148746 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:25:15.148649    2584 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 15 10:25:15.148746 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:25:15.148734    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b677254b-e134-46bb-9ea4-a6c2faeaa047-plugin-serving-cert podName:b677254b-e134-46bb-9ea4-a6c2faeaa047 nodeName:}" failed. No retries permitted until 2026-09-15 10:25:15.648710525 +0000 UTC m=+434.342866198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b677254b-e134-46bb-9ea4-a6c2faeaa047-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-dhj2d" (UID: "b677254b-e134-46bb-9ea4-a6c2faeaa047") : secret "plugin-serving-cert" not found
Sep 15 10:25:15.149078 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:15.149054    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b677254b-e134-46bb-9ea4-a6c2faeaa047-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dhj2d\" (UID: \"b677254b-e134-46bb-9ea4-a6c2faeaa047\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:15.158656 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:15.158635    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zshst\" (UniqueName: \"kubernetes.io/projected/b677254b-e134-46bb-9ea4-a6c2faeaa047-kube-api-access-zshst\") pod \"kuadrant-console-plugin-fbdd4f445-dhj2d\" (UID: \"b677254b-e134-46bb-9ea4-a6c2faeaa047\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:15.651855 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:15.651789    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b677254b-e134-46bb-9ea4-a6c2faeaa047-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dhj2d\" (UID: \"b677254b-e134-46bb-9ea4-a6c2faeaa047\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:15.654230 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:15.654203    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b677254b-e134-46bb-9ea4-a6c2faeaa047-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dhj2d\" (UID: \"b677254b-e134-46bb-9ea4-a6c2faeaa047\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:15.840938 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:15.840902    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"
Sep 15 10:25:16.109536 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:16.109501    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d"]
Sep 15 10:25:16.112558 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:25:16.112525    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb677254b_e134_46bb_9ea4_a6c2faeaa047.slice/crio-8a13539dfd63bfe0476abd7881134d98c99a1e5c8ea7fdf53a6f896771a10534 WatchSource:0}: Error finding container 8a13539dfd63bfe0476abd7881134d98c99a1e5c8ea7fdf53a6f896771a10534: Status 404 returned error can't find the container with id 8a13539dfd63bfe0476abd7881134d98c99a1e5c8ea7fdf53a6f896771a10534
Sep 15 10:25:16.534845 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:16.534789    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d" event={"ID":"b677254b-e134-46bb-9ea4-a6c2faeaa047","Type":"ContainerStarted","Data":"8a13539dfd63bfe0476abd7881134d98c99a1e5c8ea7fdf53a6f896771a10534"}
Sep 15 10:25:21.561814 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:21.561757    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d" event={"ID":"b677254b-e134-46bb-9ea4-a6c2faeaa047","Type":"ContainerStarted","Data":"d6d471373ffd8b884994b2d72bcf167e055f6df449d3465c2f6fc4fe08bdc1c1"}
Sep 15 10:25:21.578062 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:25:21.577978    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dhj2d" podStartSLOduration=2.868063815 podStartE2EDuration="7.577954943s" totalImagesPullingTime="4.709891128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:25:21.577356899 +0000 UTC m=+440.271512578" watchObservedRunningTime="2026-09-15 10:25:21.577954943 +0000 UTC m=+440.272110628"
Sep 15 10:26:11.322795 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.322756    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:26:11.326241 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.326218    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j"
Sep 15 10:26:11.328299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.328273    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 10:26:11.335301 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.335278    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:26:11.361611 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.361575    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:26:11.513436 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.513401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7239e98d-5135-49de-ac86-9979c8e37280-config-file\") pod \"limitador-limitador-84994f7ffc-k659j\" (UID: \"7239e98d-5135-49de-ac86-9979c8e37280\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j"
Sep 15 10:26:11.513598 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.513444    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psscm\" (UniqueName: \"kubernetes.io/projected/7239e98d-5135-49de-ac86-9979c8e37280-kube-api-access-psscm\") pod \"limitador-limitador-84994f7ffc-k659j\" (UID: \"7239e98d-5135-49de-ac86-9979c8e37280\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j"
Sep 15 10:26:11.614691 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.614615    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7239e98d-5135-49de-ac86-9979c8e37280-config-file\") pod \"limitador-limitador-84994f7ffc-k659j\" (UID: \"7239e98d-5135-49de-ac86-9979c8e37280\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j"
Sep 15 10:26:11.614691 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.614658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psscm\" (UniqueName: \"kubernetes.io/projected/7239e98d-5135-49de-ac86-9979c8e37280-kube-api-access-psscm\") pod \"limitador-limitador-84994f7ffc-k659j\" (UID: \"7239e98d-5135-49de-ac86-9979c8e37280\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j"
Sep 15 10:26:11.615267 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.615248    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7239e98d-5135-49de-ac86-9979c8e37280-config-file\") pod \"limitador-limitador-84994f7ffc-k659j\" (UID: \"7239e98d-5135-49de-ac86-9979c8e37280\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j"
Sep 15 10:26:11.622124 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.622103    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psscm\" (UniqueName: \"kubernetes.io/projected/7239e98d-5135-49de-ac86-9979c8e37280-kube-api-access-psscm\") pod \"limitador-limitador-84994f7ffc-k659j\" (UID: \"7239e98d-5135-49de-ac86-9979c8e37280\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j"
Sep 15 10:26:11.636916 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.636892    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j"
Sep 15 10:26:11.789867 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.789834    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljchd"]
Sep 15 10:26:11.793026 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.793006    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd"
Sep 15 10:26:11.794982 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.794960    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bcdk6\""
Sep 15 10:26:11.801362 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.801338    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljchd"]
Sep 15 10:26:11.908367 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.908294    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:26:11.910745 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.910718    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mms8"]
Sep 15 10:26:11.911606 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:26:11.911579    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7239e98d_5135_49de_ac86_9979c8e37280.slice/crio-9a20948dacb4b761492fd2920f2acfee3f129fe90c3be2c1d02a38689d2acd3a WatchSource:0}: Error finding container 9a20948dacb4b761492fd2920f2acfee3f129fe90c3be2c1d02a38689d2acd3a: Status 404 returned error can't find the container with id 9a20948dacb4b761492fd2920f2acfee3f129fe90c3be2c1d02a38689d2acd3a
Sep 15 10:26:11.913952 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.913931    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4mms8"
Sep 15 10:26:11.916778 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.916757    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8csgs\" (UniqueName: \"kubernetes.io/projected/92261a32-5d7c-4605-88a4-285c9f52913c-kube-api-access-8csgs\") pod \"authorino-7bcfd8f9cd-ljchd\" (UID: \"92261a32-5d7c-4605-88a4-285c9f52913c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd"
Sep 15 10:26:11.920884 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:11.920860    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mms8"]
Sep 15 10:26:12.017643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.017611    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8csgs\" (UniqueName: \"kubernetes.io/projected/92261a32-5d7c-4605-88a4-285c9f52913c-kube-api-access-8csgs\") pod \"authorino-7bcfd8f9cd-ljchd\" (UID: \"92261a32-5d7c-4605-88a4-285c9f52913c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd"
Sep 15 10:26:12.017643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.017650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22j4x\" (UniqueName: \"kubernetes.io/projected/b64e78fd-a7de-453d-9c83-9133d7177753-kube-api-access-22j4x\") pod \"authorino-59c974bd48-4mms8\" (UID: \"b64e78fd-a7de-453d-9c83-9133d7177753\") " pod="kuadrant-system/authorino-59c974bd48-4mms8"
Sep 15 10:26:12.025409 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.025383    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8csgs\" (UniqueName: \"kubernetes.io/projected/92261a32-5d7c-4605-88a4-285c9f52913c-kube-api-access-8csgs\") pod \"authorino-7bcfd8f9cd-ljchd\" (UID: \"92261a32-5d7c-4605-88a4-285c9f52913c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd"
Sep 15 10:26:12.103378 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.103348    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd"
Sep 15 10:26:12.118669 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.118647    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22j4x\" (UniqueName: \"kubernetes.io/projected/b64e78fd-a7de-453d-9c83-9133d7177753-kube-api-access-22j4x\") pod \"authorino-59c974bd48-4mms8\" (UID: \"b64e78fd-a7de-453d-9c83-9133d7177753\") " pod="kuadrant-system/authorino-59c974bd48-4mms8"
Sep 15 10:26:12.127096 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.127076    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22j4x\" (UniqueName: \"kubernetes.io/projected/b64e78fd-a7de-453d-9c83-9133d7177753-kube-api-access-22j4x\") pod \"authorino-59c974bd48-4mms8\" (UID: \"b64e78fd-a7de-453d-9c83-9133d7177753\") " pod="kuadrant-system/authorino-59c974bd48-4mms8"
Sep 15 10:26:12.227252 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.227183    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4mms8"
Sep 15 10:26:12.369733 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.369703    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljchd"]
Sep 15 10:26:12.372550 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:26:12.372525    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92261a32_5d7c_4605_88a4_285c9f52913c.slice/crio-5631062a8f27dc53ec4803ce76f8d0ebd40ef8551340565e65e74128f711cd4e WatchSource:0}: Error finding container 5631062a8f27dc53ec4803ce76f8d0ebd40ef8551340565e65e74128f711cd4e: Status 404 returned error can't find the container with id 5631062a8f27dc53ec4803ce76f8d0ebd40ef8551340565e65e74128f711cd4e
Sep 15 10:26:12.496509 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.496477    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mms8"]
Sep 15 10:26:12.499169 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:26:12.499132    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb64e78fd_a7de_453d_9c83_9133d7177753.slice/crio-b31826d72621a78cc13cf49953435f79525134d82f99a0499d0ddcc9e9505b9e WatchSource:0}: Error finding container b31826d72621a78cc13cf49953435f79525134d82f99a0499d0ddcc9e9505b9e: Status 404 returned error can't find the container with id b31826d72621a78cc13cf49953435f79525134d82f99a0499d0ddcc9e9505b9e
Sep 15 10:26:12.770234 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.770113    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd" event={"ID":"92261a32-5d7c-4605-88a4-285c9f52913c","Type":"ContainerStarted","Data":"5631062a8f27dc53ec4803ce76f8d0ebd40ef8551340565e65e74128f711cd4e"}
Sep 15 10:26:12.772197 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.772166    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4mms8" event={"ID":"b64e78fd-a7de-453d-9c83-9133d7177753","Type":"ContainerStarted","Data":"b31826d72621a78cc13cf49953435f79525134d82f99a0499d0ddcc9e9505b9e"}
Sep 15 10:26:12.773854 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:12.773778    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j" event={"ID":"7239e98d-5135-49de-ac86-9979c8e37280","Type":"ContainerStarted","Data":"9a20948dacb4b761492fd2920f2acfee3f129fe90c3be2c1d02a38689d2acd3a"}
Sep 15 10:26:14.785879 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:14.785837    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j" event={"ID":"7239e98d-5135-49de-ac86-9979c8e37280","Type":"ContainerStarted","Data":"136f8d85ff911b8559eaba0ff5feda525d1bc8ed7af451fd8d486c0eb31702f3"}
Sep 15 10:26:14.786365 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:14.785955    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j"
Sep 15 10:26:14.802384 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:14.802327    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j" podStartSLOduration=1.666507451 podStartE2EDuration="3.802310048s" totalImagesPullingTime="2.135802597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:26:14.800875297 +0000 UTC m=+493.495030979" watchObservedRunningTime="2026-09-15 10:26:14.802310048 +0000 UTC m=+493.496465773"
Sep 15 10:26:15.790928 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:15.790886    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd" event={"ID":"92261a32-5d7c-4605-88a4-285c9f52913c","Type":"ContainerStarted","Data":"3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3"}
Sep 15 10:26:15.792460 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:15.792432    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4mms8" event={"ID":"b64e78fd-a7de-453d-9c83-9133d7177753","Type":"ContainerStarted","Data":"6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1"}
Sep 15 10:26:15.805729 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:15.805678    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd" podStartSLOduration=1.854772775 podStartE2EDuration="4.805660751s" totalImagesPullingTime="2.950887976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:26:15.805524237 +0000 UTC m=+494.499679918" watchObservedRunningTime="2026-09-15 10:26:15.805660751 +0000 UTC m=+494.499816430"
Sep 15 10:26:15.820650 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:15.820604    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-4mms8" podStartSLOduration=2.00054864 podStartE2EDuration="4.820589892s" totalImagesPullingTime="2.820041252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:26:15.819434514 +0000 UTC m=+494.513590195" watchObservedRunningTime="2026-09-15 10:26:15.820589892 +0000 UTC m=+494.514745571"
Sep 15 10:26:15.842410 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:15.842374    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljchd"]
Sep 15 10:26:17.800457 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:17.800418    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd" podUID="92261a32-5d7c-4605-88a4-285c9f52913c" containerName="authorino" containerID="cri-o://3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3" gracePeriod=30
Sep 15 10:26:18.054342 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.054289    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd"
Sep 15 10:26:18.068283 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.068256    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8csgs\" (UniqueName: \"kubernetes.io/projected/92261a32-5d7c-4605-88a4-285c9f52913c-kube-api-access-8csgs\") pod \"92261a32-5d7c-4605-88a4-285c9f52913c\" (UID: \"92261a32-5d7c-4605-88a4-285c9f52913c\") "
Sep 15 10:26:18.070534 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.070491    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92261a32-5d7c-4605-88a4-285c9f52913c-kube-api-access-8csgs" (OuterVolumeSpecName: "kube-api-access-8csgs") pod "92261a32-5d7c-4605-88a4-285c9f52913c" (UID: "92261a32-5d7c-4605-88a4-285c9f52913c"). InnerVolumeSpecName "kube-api-access-8csgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:26:18.169178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.169150    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8csgs\" (UniqueName: \"kubernetes.io/projected/92261a32-5d7c-4605-88a4-285c9f52913c-kube-api-access-8csgs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:26:18.806730 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.806689    2584 generic.go:356] "Generic (PLEG): container finished" podID="92261a32-5d7c-4605-88a4-285c9f52913c" containerID="3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3" exitCode=0
Sep 15 10:26:18.807225 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.806747    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd"
Sep 15 10:26:18.807225 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.806775    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd" event={"ID":"92261a32-5d7c-4605-88a4-285c9f52913c","Type":"ContainerDied","Data":"3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3"}
Sep 15 10:26:18.807225 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.806838    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ljchd" event={"ID":"92261a32-5d7c-4605-88a4-285c9f52913c","Type":"ContainerDied","Data":"5631062a8f27dc53ec4803ce76f8d0ebd40ef8551340565e65e74128f711cd4e"}
Sep 15 10:26:18.807225 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.806856    2584 scope.go:117] "RemoveContainer" containerID="3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3"
Sep 15 10:26:18.816159 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.816139    2584 scope.go:117] "RemoveContainer" containerID="3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3"
Sep 15 10:26:18.816448 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:26:18.816427    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3\": container with ID starting with 3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3 not found: ID does not exist" containerID="3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3"
Sep 15 10:26:18.816495 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.816456    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3"} err="failed to get container status \"3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3\": rpc error: code = NotFound desc = could not find container \"3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3\": container with ID starting with 3d85dd8df05940c37fc27ec3926fce199731d56749eb293bd63e576b91bb6ca3 not found: ID does not exist"
Sep 15 10:26:18.827433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.827406    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljchd"]
Sep 15 10:26:18.831335 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:18.831314    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljchd"]
Sep 15 10:26:19.783742 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:19.783709    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92261a32-5d7c-4605-88a4-285c9f52913c" path="/var/lib/kubelet/pods/92261a32-5d7c-4605-88a4-285c9f52913c/volumes"
Sep 15 10:26:25.793896 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:25.793868    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-k659j"
Sep 15 10:26:43.178318 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.178287    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-29566"]
Sep 15 10:26:43.178705 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.178628    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92261a32-5d7c-4605-88a4-285c9f52913c" containerName="authorino"
Sep 15 10:26:43.178705 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.178642    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="92261a32-5d7c-4605-88a4-285c9f52913c" containerName="authorino"
Sep 15 10:26:43.178705 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.178700    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="92261a32-5d7c-4605-88a4-285c9f52913c" containerName="authorino"
Sep 15 10:26:43.181747 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.181731    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-29566"
Sep 15 10:26:43.190447 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.190424    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-29566"]
Sep 15 10:26:43.287441 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.287406    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hljpc\" (UniqueName: \"kubernetes.io/projected/cfb6e8bf-6fc5-4af8-a681-86900d27e7bd-kube-api-access-hljpc\") pod \"authorino-557f876f69-29566\" (UID: \"cfb6e8bf-6fc5-4af8-a681-86900d27e7bd\") " pod="kuadrant-system/authorino-557f876f69-29566"
Sep 15 10:26:43.388588 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.388548    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hljpc\" (UniqueName: \"kubernetes.io/projected/cfb6e8bf-6fc5-4af8-a681-86900d27e7bd-kube-api-access-hljpc\") pod \"authorino-557f876f69-29566\" (UID: \"cfb6e8bf-6fc5-4af8-a681-86900d27e7bd\") " pod="kuadrant-system/authorino-557f876f69-29566"
Sep 15 10:26:43.399371 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.399342    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hljpc\" (UniqueName: \"kubernetes.io/projected/cfb6e8bf-6fc5-4af8-a681-86900d27e7bd-kube-api-access-hljpc\") pod \"authorino-557f876f69-29566\" (UID: \"cfb6e8bf-6fc5-4af8-a681-86900d27e7bd\") " pod="kuadrant-system/authorino-557f876f69-29566"
Sep 15 10:26:43.411211 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.411179    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-29566"]
Sep 15 10:26:43.411453 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.411440    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-29566"
Sep 15 10:26:43.439151 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.439086    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85954544f9-qdwf8"]
Sep 15 10:26:43.442946 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.442926    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85954544f9-qdwf8"
Sep 15 10:26:43.446682 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.446658    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85954544f9-qdwf8"]
Sep 15 10:26:43.589849 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.589784    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2d7\" (UniqueName: \"kubernetes.io/projected/452d9dd8-6aaf-421f-8763-ba192185cd93-kube-api-access-6w2d7\") pod \"authorino-85954544f9-qdwf8\" (UID: \"452d9dd8-6aaf-421f-8763-ba192185cd93\") " pod="kuadrant-system/authorino-85954544f9-qdwf8"
Sep 15 10:26:43.683543 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.683387    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-29566"]
Sep 15 10:26:43.686261 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:26:43.686230    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfb6e8bf_6fc5_4af8_a681_86900d27e7bd.slice/crio-0fc5746d48dc291756e3d96a40429195ed66d1b1443473db5918ded6af89c7f9 WatchSource:0}: Error finding container 0fc5746d48dc291756e3d96a40429195ed66d1b1443473db5918ded6af89c7f9: Status 404 returned error can't find the container with id 0fc5746d48dc291756e3d96a40429195ed66d1b1443473db5918ded6af89c7f9
Sep 15 10:26:43.690459 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.690395    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2d7\" (UniqueName: \"kubernetes.io/projected/452d9dd8-6aaf-421f-8763-ba192185cd93-kube-api-access-6w2d7\") pod \"authorino-85954544f9-qdwf8\" (UID: \"452d9dd8-6aaf-421f-8763-ba192185cd93\") " pod="kuadrant-system/authorino-85954544f9-qdwf8"
Sep 15 10:26:43.698200 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.698172    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2d7\" (UniqueName: \"kubernetes.io/projected/452d9dd8-6aaf-421f-8763-ba192185cd93-kube-api-access-6w2d7\") pod \"authorino-85954544f9-qdwf8\" (UID: \"452d9dd8-6aaf-421f-8763-ba192185cd93\") " pod="kuadrant-system/authorino-85954544f9-qdwf8"
Sep 15 10:26:43.755042 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.755020    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85954544f9-qdwf8"
Sep 15 10:26:43.908844 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:43.908787    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-29566" event={"ID":"cfb6e8bf-6fc5-4af8-a681-86900d27e7bd","Type":"ContainerStarted","Data":"0fc5746d48dc291756e3d96a40429195ed66d1b1443473db5918ded6af89c7f9"}
Sep 15 10:26:44.022211 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:44.022176    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85954544f9-qdwf8"]
Sep 15 10:26:44.024897 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:26:44.024870    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod452d9dd8_6aaf_421f_8763_ba192185cd93.slice/crio-8c6c7bb4d88c18d42a9991090850d633b65b4670380ab34b9be74f6ac1675491 WatchSource:0}: Error finding container 8c6c7bb4d88c18d42a9991090850d633b65b4670380ab34b9be74f6ac1675491: Status 404 returned error can't find the container with id 8c6c7bb4d88c18d42a9991090850d633b65b4670380ab34b9be74f6ac1675491
Sep 15 10:26:44.915167 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:44.915070    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85954544f9-qdwf8" event={"ID":"452d9dd8-6aaf-421f-8763-ba192185cd93","Type":"ContainerStarted","Data":"4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e"}
Sep 15 10:26:44.915167 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:44.915121    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85954544f9-qdwf8" event={"ID":"452d9dd8-6aaf-421f-8763-ba192185cd93","Type":"ContainerStarted","Data":"8c6c7bb4d88c18d42a9991090850d633b65b4670380ab34b9be74f6ac1675491"}
Sep 15 10:26:44.916586 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:44.916560    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-29566" event={"ID":"cfb6e8bf-6fc5-4af8-a681-86900d27e7bd","Type":"ContainerStarted","Data":"79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7"}
Sep 15 10:26:44.916706 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:44.916675    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-29566" podUID="cfb6e8bf-6fc5-4af8-a681-86900d27e7bd" containerName="authorino" containerID="cri-o://79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7" gracePeriod=30
Sep 15 10:26:44.929038 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:44.928921    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85954544f9-qdwf8" podStartSLOduration=1.294733307 podStartE2EDuration="1.928906701s" totalImagesPullingTime="634.173394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:26:44.928167806 +0000 UTC m=+523.622323500" watchObservedRunningTime="2026-09-15 10:26:44.928906701 +0000 UTC m=+523.623062380"
Sep 15 10:26:44.943368 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:44.943327    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-29566" podStartSLOduration=1.429823951 podStartE2EDuration="1.943313046s" totalImagesPullingTime="513.489095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:26:44.942442027 +0000 UTC m=+523.636597710" watchObservedRunningTime="2026-09-15 10:26:44.943313046 +0000 UTC m=+523.637468725"
Sep 15 10:26:44.957138 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:44.957109    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mms8"]
Sep 15 10:26:44.957321 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:44.957299    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-4mms8" podUID="b64e78fd-a7de-453d-9c83-9133d7177753" containerName="authorino" containerID="cri-o://6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1" gracePeriod=30
Sep 15 10:26:45.029537 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.029508    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d6985549c-ckskt"]
Sep 15 10:26:45.032566 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.032546    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6985549c-ckskt"
Sep 15 10:26:45.034737 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.034710    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 10:26:45.039661 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.039633    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d6985549c-ckskt"]
Sep 15 10:26:45.100830 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.100688    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-tls-cert\") pod \"authorino-6d6985549c-ckskt\" (UID: \"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16\") " pod="kuadrant-system/authorino-6d6985549c-ckskt"
Sep 15 10:26:45.100830 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.100761    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6j6r\" (UniqueName: \"kubernetes.io/projected/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-kube-api-access-m6j6r\") pod \"authorino-6d6985549c-ckskt\" (UID: \"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16\") " pod="kuadrant-system/authorino-6d6985549c-ckskt"
Sep 15 10:26:45.201344 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.201257    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6j6r\" (UniqueName: \"kubernetes.io/projected/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-kube-api-access-m6j6r\") pod \"authorino-6d6985549c-ckskt\" (UID: \"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16\") " pod="kuadrant-system/authorino-6d6985549c-ckskt"
Sep 15 10:26:45.201344 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.201308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-tls-cert\") pod \"authorino-6d6985549c-ckskt\" (UID: \"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16\") " pod="kuadrant-system/authorino-6d6985549c-ckskt"
Sep 15 10:26:45.203396 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.203372    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-29566"
Sep 15 10:26:45.203676 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.203645    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-tls-cert\") pod \"authorino-6d6985549c-ckskt\" (UID: \"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16\") " pod="kuadrant-system/authorino-6d6985549c-ckskt"
Sep 15 10:26:45.208080 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.208059    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6j6r\" (UniqueName: \"kubernetes.io/projected/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-kube-api-access-m6j6r\") pod \"authorino-6d6985549c-ckskt\" (UID: \"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16\") " pod="kuadrant-system/authorino-6d6985549c-ckskt"
Sep 15 10:26:45.208171 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.208089    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4mms8"
Sep 15 10:26:45.301865 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.301833    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hljpc\" (UniqueName: \"kubernetes.io/projected/cfb6e8bf-6fc5-4af8-a681-86900d27e7bd-kube-api-access-hljpc\") pod \"cfb6e8bf-6fc5-4af8-a681-86900d27e7bd\" (UID: \"cfb6e8bf-6fc5-4af8-a681-86900d27e7bd\") "
Sep 15 10:26:45.302030 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.301938    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22j4x\" (UniqueName: \"kubernetes.io/projected/b64e78fd-a7de-453d-9c83-9133d7177753-kube-api-access-22j4x\") pod \"b64e78fd-a7de-453d-9c83-9133d7177753\" (UID: \"b64e78fd-a7de-453d-9c83-9133d7177753\") "
Sep 15 10:26:45.303945 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.303905    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64e78fd-a7de-453d-9c83-9133d7177753-kube-api-access-22j4x" (OuterVolumeSpecName: "kube-api-access-22j4x") pod "b64e78fd-a7de-453d-9c83-9133d7177753" (UID: "b64e78fd-a7de-453d-9c83-9133d7177753"). InnerVolumeSpecName "kube-api-access-22j4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:26:45.303945 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.303914    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb6e8bf-6fc5-4af8-a681-86900d27e7bd-kube-api-access-hljpc" (OuterVolumeSpecName: "kube-api-access-hljpc") pod "cfb6e8bf-6fc5-4af8-a681-86900d27e7bd" (UID: "cfb6e8bf-6fc5-4af8-a681-86900d27e7bd"). InnerVolumeSpecName "kube-api-access-hljpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:26:45.341271 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.341232    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6985549c-ckskt"
Sep 15 10:26:45.403670 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.403599    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22j4x\" (UniqueName: \"kubernetes.io/projected/b64e78fd-a7de-453d-9c83-9133d7177753-kube-api-access-22j4x\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:26:45.403670 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.403635    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hljpc\" (UniqueName: \"kubernetes.io/projected/cfb6e8bf-6fc5-4af8-a681-86900d27e7bd-kube-api-access-hljpc\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:26:45.612207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.612174    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d6985549c-ckskt"]
Sep 15 10:26:45.614949 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:26:45.614919    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf88ca6c3_af02_4c6e_8cf8_3f086a5d2f16.slice/crio-3d9b06640df2da955d8c99e71f6639c5e7ed33cab90c534832f1fb4f1d35807a WatchSource:0}: Error finding container 3d9b06640df2da955d8c99e71f6639c5e7ed33cab90c534832f1fb4f1d35807a: Status 404 returned error can't find the container with id 3d9b06640df2da955d8c99e71f6639c5e7ed33cab90c534832f1fb4f1d35807a
Sep 15 10:26:45.798117 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:26:45.798052    2584 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfb6e8bf_6fc5_4af8_a681_86900d27e7bd.slice\": RecentStats: unable to find data in memory cache]"
Sep 15 10:26:45.922880 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.922784    2584 generic.go:356] "Generic (PLEG): container finished" podID="cfb6e8bf-6fc5-4af8-a681-86900d27e7bd" containerID="79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7" exitCode=0
Sep 15 10:26:45.922880 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.922837    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-29566"
Sep 15 10:26:45.922880 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.922869    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-29566" event={"ID":"cfb6e8bf-6fc5-4af8-a681-86900d27e7bd","Type":"ContainerDied","Data":"79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7"}
Sep 15 10:26:45.923362 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.922907    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-29566" event={"ID":"cfb6e8bf-6fc5-4af8-a681-86900d27e7bd","Type":"ContainerDied","Data":"0fc5746d48dc291756e3d96a40429195ed66d1b1443473db5918ded6af89c7f9"}
Sep 15 10:26:45.923362 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.922923    2584 scope.go:117] "RemoveContainer" containerID="79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7"
Sep 15 10:26:45.929844 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.929642    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6985549c-ckskt" event={"ID":"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16","Type":"ContainerStarted","Data":"3d9b06640df2da955d8c99e71f6639c5e7ed33cab90c534832f1fb4f1d35807a"}
Sep 15 10:26:45.931264 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.931237    2584 generic.go:356] "Generic (PLEG): container finished" podID="b64e78fd-a7de-453d-9c83-9133d7177753" containerID="6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1" exitCode=0
Sep 15 10:26:45.931396 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.931286    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4mms8"
Sep 15 10:26:45.931396 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.931313    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4mms8" event={"ID":"b64e78fd-a7de-453d-9c83-9133d7177753","Type":"ContainerDied","Data":"6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1"}
Sep 15 10:26:45.931396 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.931356    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4mms8" event={"ID":"b64e78fd-a7de-453d-9c83-9133d7177753","Type":"ContainerDied","Data":"b31826d72621a78cc13cf49953435f79525134d82f99a0499d0ddcc9e9505b9e"}
Sep 15 10:26:45.936620 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.936605    2584 scope.go:117] "RemoveContainer" containerID="79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7"
Sep 15 10:26:45.936893 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:26:45.936859    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7\": container with ID starting with 79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7 not found: ID does not exist" containerID="79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7"
Sep 15 10:26:45.936980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.936889    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7"} err="failed to get container status \"79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7\": rpc error: code = NotFound desc = could not find container \"79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7\": container with ID starting with 79f6baad49c31e9284702be57067356c20ce6826d80d5357739386296bd147d7 not found: ID does not exist"
Sep 15 10:26:45.936980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.936906    2584 scope.go:117] "RemoveContainer" containerID="6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1"
Sep 15 10:26:45.942405 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.942386    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-29566"]
Sep 15 10:26:45.943921 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.943902    2584 scope.go:117] "RemoveContainer" containerID="6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1"
Sep 15 10:26:45.944171 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:26:45.944147    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1\": container with ID starting with 6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1 not found: ID does not exist" containerID="6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1"
Sep 15 10:26:45.944272 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.944181    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1"} err="failed to get container status \"6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1\": rpc error: code = NotFound desc = could not find container \"6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1\": container with ID starting with 6c5fe85cb00abe833bff640385b6f929360dad170bc671f921a237699157feb1 not found: ID does not exist"
Sep 15 10:26:45.946085 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.946065    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-29566"]
Sep 15 10:26:45.954281 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.954262    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mms8"]
Sep 15 10:26:45.956101 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:45.956081    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mms8"]
Sep 15 10:26:46.938932 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:46.938896    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6985549c-ckskt" event={"ID":"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16","Type":"ContainerStarted","Data":"161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88"}
Sep 15 10:26:46.953869 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:46.953823    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d6985549c-ckskt" podStartSLOduration=1.457662912 podStartE2EDuration="1.953790393s" totalImagesPullingTime="496.127481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:26:46.95315059 +0000 UTC m=+525.647306271" watchObservedRunningTime="2026-09-15 10:26:46.953790393 +0000 UTC m=+525.647946115"
Sep 15 10:26:46.980518 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:46.980482    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85954544f9-qdwf8"]
Sep 15 10:26:46.980680 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:46.980642    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-85954544f9-qdwf8" podUID="452d9dd8-6aaf-421f-8763-ba192185cd93" containerName="authorino" containerID="cri-o://4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e" gracePeriod=30
Sep 15 10:26:47.219068 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.219047    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85954544f9-qdwf8"
Sep 15 10:26:47.320009 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.319974    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w2d7\" (UniqueName: \"kubernetes.io/projected/452d9dd8-6aaf-421f-8763-ba192185cd93-kube-api-access-6w2d7\") pod \"452d9dd8-6aaf-421f-8763-ba192185cd93\" (UID: \"452d9dd8-6aaf-421f-8763-ba192185cd93\") "
Sep 15 10:26:47.322203 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.322164    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452d9dd8-6aaf-421f-8763-ba192185cd93-kube-api-access-6w2d7" (OuterVolumeSpecName: "kube-api-access-6w2d7") pod "452d9dd8-6aaf-421f-8763-ba192185cd93" (UID: "452d9dd8-6aaf-421f-8763-ba192185cd93"). InnerVolumeSpecName "kube-api-access-6w2d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:26:47.426640 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.426600    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w2d7\" (UniqueName: \"kubernetes.io/projected/452d9dd8-6aaf-421f-8763-ba192185cd93-kube-api-access-6w2d7\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:26:47.785360 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.785325    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b64e78fd-a7de-453d-9c83-9133d7177753" path="/var/lib/kubelet/pods/b64e78fd-a7de-453d-9c83-9133d7177753/volumes"
Sep 15 10:26:47.785654 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.785641    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfb6e8bf-6fc5-4af8-a681-86900d27e7bd" path="/var/lib/kubelet/pods/cfb6e8bf-6fc5-4af8-a681-86900d27e7bd/volumes"
Sep 15 10:26:47.943745 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.943712    2584 generic.go:356] "Generic (PLEG): container finished" podID="452d9dd8-6aaf-421f-8763-ba192185cd93" containerID="4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e" exitCode=0
Sep 15 10:26:47.943745 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.943751    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85954544f9-qdwf8"
Sep 15 10:26:47.944220 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.943764    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85954544f9-qdwf8" event={"ID":"452d9dd8-6aaf-421f-8763-ba192185cd93","Type":"ContainerDied","Data":"4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e"}
Sep 15 10:26:47.944220 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.943823    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85954544f9-qdwf8" event={"ID":"452d9dd8-6aaf-421f-8763-ba192185cd93","Type":"ContainerDied","Data":"8c6c7bb4d88c18d42a9991090850d633b65b4670380ab34b9be74f6ac1675491"}
Sep 15 10:26:47.944220 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.943844    2584 scope.go:117] "RemoveContainer" containerID="4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e"
Sep 15 10:26:47.953064 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.953045    2584 scope.go:117] "RemoveContainer" containerID="4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e"
Sep 15 10:26:47.953350 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:26:47.953328    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e\": container with ID starting with 4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e not found: ID does not exist" containerID="4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e"
Sep 15 10:26:47.953438 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.953357    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e"} err="failed to get container status \"4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e\": rpc error: code = NotFound desc = could not find container \"4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e\": container with ID starting with 4a0b3c4ece225f70cd6b312c165daf25bef92628d7cee0f28425f2b9682bfa1e not found: ID does not exist"
Sep 15 10:26:47.958172 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.958152    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85954544f9-qdwf8"]
Sep 15 10:26:47.961665 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:47.961645    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-85954544f9-qdwf8"]
Sep 15 10:26:49.784735 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:26:49.784703    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="452d9dd8-6aaf-421f-8763-ba192185cd93" path="/var/lib/kubelet/pods/452d9dd8-6aaf-421f-8763-ba192185cd93/volumes"
Sep 15 10:27:26.491381 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:27:26.491336    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:27:59.559871 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:27:59.559837    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:28:05.656936 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:05.656889    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:28:12.058351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:12.058307    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:28:22.857151 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.857117    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"]
Sep 15 10:28:22.857610 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.857441    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452d9dd8-6aaf-421f-8763-ba192185cd93" containerName="authorino"
Sep 15 10:28:22.857610 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.857454    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="452d9dd8-6aaf-421f-8763-ba192185cd93" containerName="authorino"
Sep 15 10:28:22.857610 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.857471    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb6e8bf-6fc5-4af8-a681-86900d27e7bd" containerName="authorino"
Sep 15 10:28:22.857610 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.857479    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb6e8bf-6fc5-4af8-a681-86900d27e7bd" containerName="authorino"
Sep 15 10:28:22.857610 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.857502    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b64e78fd-a7de-453d-9c83-9133d7177753" containerName="authorino"
Sep 15 10:28:22.857610 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.857511    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b64e78fd-a7de-453d-9c83-9133d7177753" containerName="authorino"
Sep 15 10:28:22.858008 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.857620    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b64e78fd-a7de-453d-9c83-9133d7177753" containerName="authorino"
Sep 15 10:28:22.858008 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.857632    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb6e8bf-6fc5-4af8-a681-86900d27e7bd" containerName="authorino"
Sep 15 10:28:22.858008 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.857638    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="452d9dd8-6aaf-421f-8763-ba192185cd93" containerName="authorino"
Sep 15 10:28:22.860643 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.860624    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:22.862929 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.862903    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 10:28:22.863057 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.862977    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 15 10:28:22.863930 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.863908    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 10:28:22.864273 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.864249    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fwljq\""
Sep 15 10:28:22.868686 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.868664    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"]
Sep 15 10:28:22.914473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.914433    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:22.914473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.914471    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:22.914686 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.914499    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:22.914686 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.914562    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:22.914686 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.914648    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2877d7e7-14ea-4a16-bee1-08281728d61f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:22.914686 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:22.914680    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5l4\" (UniqueName: \"kubernetes.io/projected/2877d7e7-14ea-4a16-bee1-08281728d61f-kube-api-access-nl5l4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.015245 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.015194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2877d7e7-14ea-4a16-bee1-08281728d61f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.015245 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.015254    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5l4\" (UniqueName: \"kubernetes.io/projected/2877d7e7-14ea-4a16-bee1-08281728d61f-kube-api-access-nl5l4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.015523 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.015285    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.015523 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.015306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.015523 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.015330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.015523 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.015355    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.015784 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.015747    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.015932 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.015761    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.015932 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.015828    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.017575 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.017553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2877d7e7-14ea-4a16-bee1-08281728d61f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.017680 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.017661    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2877d7e7-14ea-4a16-bee1-08281728d61f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.022880 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.022858    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5l4\" (UniqueName: \"kubernetes.io/projected/2877d7e7-14ea-4a16-bee1-08281728d61f-kube-api-access-nl5l4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5wn4g\" (UID: \"2877d7e7-14ea-4a16-bee1-08281728d61f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.173509 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.173415    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:23.443057 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.443025    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"]
Sep 15 10:28:23.445435 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:28:23.445407    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2877d7e7_14ea_4a16_bee1_08281728d61f.slice/crio-ac2707d9e5123a5dc77f527575fe33ce4fc1d30d5c239bacb97c1f8e6db505a0 WatchSource:0}: Error finding container ac2707d9e5123a5dc77f527575fe33ce4fc1d30d5c239bacb97c1f8e6db505a0: Status 404 returned error can't find the container with id ac2707d9e5123a5dc77f527575fe33ce4fc1d30d5c239bacb97c1f8e6db505a0
Sep 15 10:28:23.446710 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:23.446692    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 10:28:24.060775 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:24.060744    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:28:24.345185 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:24.345100    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g" event={"ID":"2877d7e7-14ea-4a16-bee1-08281728d61f","Type":"ContainerStarted","Data":"ac2707d9e5123a5dc77f527575fe33ce4fc1d30d5c239bacb97c1f8e6db505a0"}
Sep 15 10:28:26.355019 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:26.354977    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g" event={"ID":"2877d7e7-14ea-4a16-bee1-08281728d61f","Type":"ContainerStarted","Data":"5da877855ac9f45bfce7a73904371c2b0d26a661a9e5728a5f7cae1972bd5afe"}
Sep 15 10:28:26.370789 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:26.370731    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g" podStartSLOduration=2.394561627 podStartE2EDuration="4.370713987s" totalImagesPullingTime="1.97615236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:28:26.369903861 +0000 UTC m=+625.064059542" watchObservedRunningTime="2026-09-15 10:28:26.370713987 +0000 UTC m=+625.064869667"
Sep 15 10:28:27.153952 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.151983    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"]
Sep 15 10:28:27.159492 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.159464    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.162990 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.162962    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 15 10:28:27.164326 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.164289    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"]
Sep 15 10:28:27.260357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.260317    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.260543 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.260364    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.260543 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.260427    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dbfaf81-5967-4d80-a60e-b069ccb75342-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.260543 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.260451    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.260543 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.260502    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.260543 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.260533    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zxxk\" (UniqueName: \"kubernetes.io/projected/1dbfaf81-5967-4d80-a60e-b069ccb75342-kube-api-access-8zxxk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.361514 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.361484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.361978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.361526    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.361978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.361669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dbfaf81-5967-4d80-a60e-b069ccb75342-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.361978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.361711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.361978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.361751    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.361978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.361783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zxxk\" (UniqueName: \"kubernetes.io/projected/1dbfaf81-5967-4d80-a60e-b069ccb75342-kube-api-access-8zxxk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.361978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.361909    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.361978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.361952    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.362302 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.362073    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.364003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.363979    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dbfaf81-5967-4d80-a60e-b069ccb75342-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.364288 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.364269    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dbfaf81-5967-4d80-a60e-b069ccb75342-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.369573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.369549    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zxxk\" (UniqueName: \"kubernetes.io/projected/1dbfaf81-5967-4d80-a60e-b069ccb75342-kube-api-access-8zxxk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4\" (UID: \"1dbfaf81-5967-4d80-a60e-b069ccb75342\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.472467 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.472389    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:27.740924 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:27.740897    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"]
Sep 15 10:28:27.743211 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:28:27.743171    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbfaf81_5967_4d80_a60e_b069ccb75342.slice/crio-0059af76cbce65d378bf13dcfd20f6d38026144fed4064f0577976e0c81764ce WatchSource:0}: Error finding container 0059af76cbce65d378bf13dcfd20f6d38026144fed4064f0577976e0c81764ce: Status 404 returned error can't find the container with id 0059af76cbce65d378bf13dcfd20f6d38026144fed4064f0577976e0c81764ce
Sep 15 10:28:28.361725 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:28.361684    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:28:28.365490 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:28.365463    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4" event={"ID":"1dbfaf81-5967-4d80-a60e-b069ccb75342","Type":"ContainerStarted","Data":"99dbdc7f4630fd270d961a74f04ebb2c0f975b044f9d1a3129217a42580b2e8d"}
Sep 15 10:28:28.365654 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:28.365507    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4" event={"ID":"1dbfaf81-5967-4d80-a60e-b069ccb75342","Type":"ContainerStarted","Data":"0059af76cbce65d378bf13dcfd20f6d38026144fed4064f0577976e0c81764ce"}
Sep 15 10:28:28.383218 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:28.383170    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4" podStartSLOduration=1.1957171149999999 podStartE2EDuration="1.38315504s" totalImagesPullingTime="187.437925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:28:28.381177163 +0000 UTC m=+627.075332843" watchObservedRunningTime="2026-09-15 10:28:28.38315504 +0000 UTC m=+627.077310719"
Sep 15 10:28:33.173796 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:33.173759    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:33.174253 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:33.173874    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:33.186122 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:33.186100    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:33.396323 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:33.396291    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5wn4g"
Sep 15 10:28:37.473282 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:37.473250    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:37.473778 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:37.473313    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:37.485681 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:37.485651    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:38.416185 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:38.416153    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4"
Sep 15 10:28:43.359142 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:43.359101    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:28:56.738889 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.738854    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"]
Sep 15 10:28:56.742092 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.742075    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.744469 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.744437    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 10:28:56.752159 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.752136    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"]
Sep 15 10:28:56.817518 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.817479    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntnrm\" (UniqueName: \"kubernetes.io/projected/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-kube-api-access-ntnrm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.817518 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.817522    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.817731 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.817579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.817731 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.817613    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.817731 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.817670    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.817731 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.817713    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.919003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.918966    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.919193 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.919016    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.919193 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.919111    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntnrm\" (UniqueName: \"kubernetes.io/projected/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-kube-api-access-ntnrm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.919193 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.919142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.919193 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.919167    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.919401 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.919192    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.919541 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.919515    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.919541 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.919535    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.919642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.919562    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.921322 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.921299    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.921511 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.921492    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:56.926833 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:56.926812    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntnrm\" (UniqueName: \"kubernetes.io/projected/71dbe841-0517-42f6-bdea-c0ddbf9ec3a7-kube-api-access-ntnrm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b68lp\" (UID: \"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:57.053046 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:57.052994    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:28:57.321936 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:57.321907    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"]
Sep 15 10:28:57.324487 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:28:57.324463    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71dbe841_0517_42f6_bdea_c0ddbf9ec3a7.slice/crio-21f6a2e2621a2194cadc3bc076c299174e47b39f41c98ae9297c43e8ab2d9f72 WatchSource:0}: Error finding container 21f6a2e2621a2194cadc3bc076c299174e47b39f41c98ae9297c43e8ab2d9f72: Status 404 returned error can't find the container with id 21f6a2e2621a2194cadc3bc076c299174e47b39f41c98ae9297c43e8ab2d9f72
Sep 15 10:28:57.487276 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:57.487245    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp" event={"ID":"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7","Type":"ContainerStarted","Data":"21f6a2e2621a2194cadc3bc076c299174e47b39f41c98ae9297c43e8ab2d9f72"}
Sep 15 10:28:58.493441 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:58.493407    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp" event={"ID":"71dbe841-0517-42f6-bdea-c0ddbf9ec3a7","Type":"ContainerStarted","Data":"ab754fe04beaefa5663b225b5ed2c53d7bd4f750485ccc7c37d816bf2b7bbafc"}
Sep 15 10:28:58.510498 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:58.510436    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp" podStartSLOduration=2.29663962 podStartE2EDuration="2.510419203s" totalImagesPullingTime="213.779583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:28:58.510340209 +0000 UTC m=+657.204495890" watchObservedRunningTime="2026-09-15 10:28:58.510419203 +0000 UTC m=+657.204574883"
Sep 15 10:28:59.162367 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:28:59.162323    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:29:07.053353 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:07.053321    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:29:07.053778 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:07.053386    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:29:07.065781 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:07.065757    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:29:07.541816 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:07.541776    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b68lp"
Sep 15 10:29:54.708733 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:54.708651    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68ccdd49b4-pqt29"]
Sep 15 10:29:54.712248 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:54.712229    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68ccdd49b4-pqt29"
Sep 15 10:29:54.721532 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:54.721509    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68ccdd49b4-pqt29"]
Sep 15 10:29:54.839386 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:54.839355    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fda4c709-2973-480b-8a9b-7cfe35c77070-tls-cert\") pod \"authorino-68ccdd49b4-pqt29\" (UID: \"fda4c709-2973-480b-8a9b-7cfe35c77070\") " pod="kuadrant-system/authorino-68ccdd49b4-pqt29"
Sep 15 10:29:54.839557 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:54.839400    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5dcf\" (UniqueName: \"kubernetes.io/projected/fda4c709-2973-480b-8a9b-7cfe35c77070-kube-api-access-p5dcf\") pod \"authorino-68ccdd49b4-pqt29\" (UID: \"fda4c709-2973-480b-8a9b-7cfe35c77070\") " pod="kuadrant-system/authorino-68ccdd49b4-pqt29"
Sep 15 10:29:54.940633 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:54.940599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fda4c709-2973-480b-8a9b-7cfe35c77070-tls-cert\") pod \"authorino-68ccdd49b4-pqt29\" (UID: \"fda4c709-2973-480b-8a9b-7cfe35c77070\") " pod="kuadrant-system/authorino-68ccdd49b4-pqt29"
Sep 15 10:29:54.940768 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:54.940645    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5dcf\" (UniqueName: \"kubernetes.io/projected/fda4c709-2973-480b-8a9b-7cfe35c77070-kube-api-access-p5dcf\") pod \"authorino-68ccdd49b4-pqt29\" (UID: \"fda4c709-2973-480b-8a9b-7cfe35c77070\") " pod="kuadrant-system/authorino-68ccdd49b4-pqt29"
Sep 15 10:29:54.942978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:54.942958    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fda4c709-2973-480b-8a9b-7cfe35c77070-tls-cert\") pod \"authorino-68ccdd49b4-pqt29\" (UID: \"fda4c709-2973-480b-8a9b-7cfe35c77070\") " pod="kuadrant-system/authorino-68ccdd49b4-pqt29"
Sep 15 10:29:54.948241 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:54.948219    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5dcf\" (UniqueName: \"kubernetes.io/projected/fda4c709-2973-480b-8a9b-7cfe35c77070-kube-api-access-p5dcf\") pod \"authorino-68ccdd49b4-pqt29\" (UID: \"fda4c709-2973-480b-8a9b-7cfe35c77070\") " pod="kuadrant-system/authorino-68ccdd49b4-pqt29"
Sep 15 10:29:55.020950 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:55.020918    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68ccdd49b4-pqt29"
Sep 15 10:29:55.289249 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:55.289221    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68ccdd49b4-pqt29"]
Sep 15 10:29:55.291921 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:29:55.291890    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfda4c709_2973_480b_8a9b_7cfe35c77070.slice/crio-6dac0e171f5c707756137562c281d0e06ca1ec5757d6d6a350662ed0605ce406 WatchSource:0}: Error finding container 6dac0e171f5c707756137562c281d0e06ca1ec5757d6d6a350662ed0605ce406: Status 404 returned error can't find the container with id 6dac0e171f5c707756137562c281d0e06ca1ec5757d6d6a350662ed0605ce406
Sep 15 10:29:55.737105 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:55.737022    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68ccdd49b4-pqt29" event={"ID":"fda4c709-2973-480b-8a9b-7cfe35c77070","Type":"ContainerStarted","Data":"6dac0e171f5c707756137562c281d0e06ca1ec5757d6d6a350662ed0605ce406"}
Sep 15 10:29:56.744016 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:56.743976    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68ccdd49b4-pqt29" event={"ID":"fda4c709-2973-480b-8a9b-7cfe35c77070","Type":"ContainerStarted","Data":"488efc740f7589cd8b7c1e2ffae833e6a938fa6559314a109b9ab2ac0a799a38"}
Sep 15 10:29:56.760206 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:56.760151    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-68ccdd49b4-pqt29" podStartSLOduration=1.99348529 podStartE2EDuration="2.760135279s" totalImagesPullingTime="766.649989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:29:56.758454595 +0000 UTC m=+715.452610275" watchObservedRunningTime="2026-09-15 10:29:56.760135279 +0000 UTC m=+715.454290960"
Sep 15 10:29:56.791321 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:56.790631    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d6985549c-ckskt"]
Sep 15 10:29:56.791321 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:56.790871    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d6985549c-ckskt" podUID="f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16" containerName="authorino" containerID="cri-o://161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88" gracePeriod=30
Sep 15 10:29:57.038366 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.038339    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6985549c-ckskt"
Sep 15 10:29:57.161982 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.161942    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-tls-cert\") pod \"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16\" (UID: \"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16\") "
Sep 15 10:29:57.162148 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.161992    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6j6r\" (UniqueName: \"kubernetes.io/projected/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-kube-api-access-m6j6r\") pod \"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16\" (UID: \"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16\") "
Sep 15 10:29:57.163961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.163926    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-kube-api-access-m6j6r" (OuterVolumeSpecName: "kube-api-access-m6j6r") pod "f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16" (UID: "f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16"). InnerVolumeSpecName "kube-api-access-m6j6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:29:57.171086 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.171043    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16" (UID: "f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:29:57.263060 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.263019    2584 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-tls-cert\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:29:57.263060 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.263055    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6j6r\" (UniqueName: \"kubernetes.io/projected/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16-kube-api-access-m6j6r\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:29:57.749791 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.749745    2584 generic.go:356] "Generic (PLEG): container finished" podID="f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16" containerID="161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88" exitCode=0
Sep 15 10:29:57.750335 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.749819    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6985549c-ckskt"
Sep 15 10:29:57.750335 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.749834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6985549c-ckskt" event={"ID":"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16","Type":"ContainerDied","Data":"161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88"}
Sep 15 10:29:57.750335 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.749881    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6985549c-ckskt" event={"ID":"f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16","Type":"ContainerDied","Data":"3d9b06640df2da955d8c99e71f6639c5e7ed33cab90c534832f1fb4f1d35807a"}
Sep 15 10:29:57.750335 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.749900    2584 scope.go:117] "RemoveContainer" containerID="161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88"
Sep 15 10:29:57.759322 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.759305    2584 scope.go:117] "RemoveContainer" containerID="161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88"
Sep 15 10:29:57.759629 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:29:57.759607    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88\": container with ID starting with 161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88 not found: ID does not exist" containerID="161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88"
Sep 15 10:29:57.759717 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.759636    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88"} err="failed to get container status \"161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88\": rpc error: code = NotFound desc = could not find container \"161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88\": container with ID starting with 161b6364d557f2cd0308a66d15e2f0476adaf942292f7bc96ec9d4546428ad88 not found: ID does not exist"
Sep 15 10:29:57.770392 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.770362    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d6985549c-ckskt"]
Sep 15 10:29:57.774130 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.774103    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d6985549c-ckskt"]
Sep 15 10:29:57.785442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:29:57.785409    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16" path="/var/lib/kubelet/pods/f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16/volumes"
Sep 15 10:30:16.661528 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:16.661493    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7891-53fdce"]
Sep 15 10:30:16.662059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:16.661826    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16" containerName="authorino"
Sep 15 10:30:16.662059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:16.661837    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16" containerName="authorino"
Sep 15 10:30:16.662059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:16.661897    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f88ca6c3-af02-4c6e-8cf8-3f086a5d2f16" containerName="authorino"
Sep 15 10:30:16.666207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:16.666190    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-53fdce"
Sep 15 10:30:16.668456 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:16.668433    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f7bvs\""
Sep 15 10:30:16.675250 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:16.674328    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7891-53fdce"]
Sep 15 10:30:16.730735 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:16.730702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k4b2\" (UniqueName: \"kubernetes.io/projected/c0e23997-ca08-48ae-b74c-f2a97f47ca2d-kube-api-access-4k4b2\") pod \"test-curl-7891-53fdce\" (UID: \"c0e23997-ca08-48ae-b74c-f2a97f47ca2d\") " pod="opendatahub/test-curl-7891-53fdce"
Sep 15 10:30:16.831238 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:16.831199    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k4b2\" (UniqueName: \"kubernetes.io/projected/c0e23997-ca08-48ae-b74c-f2a97f47ca2d-kube-api-access-4k4b2\") pod \"test-curl-7891-53fdce\" (UID: \"c0e23997-ca08-48ae-b74c-f2a97f47ca2d\") " pod="opendatahub/test-curl-7891-53fdce"
Sep 15 10:30:16.838686 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:16.838653    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k4b2\" (UniqueName: \"kubernetes.io/projected/c0e23997-ca08-48ae-b74c-f2a97f47ca2d-kube-api-access-4k4b2\") pod \"test-curl-7891-53fdce\" (UID: \"c0e23997-ca08-48ae-b74c-f2a97f47ca2d\") " pod="opendatahub/test-curl-7891-53fdce"
Sep 15 10:30:16.976268 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:16.976182    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-53fdce"
Sep 15 10:30:17.240994 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:17.240914    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7891-53fdce"]
Sep 15 10:30:17.244086 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:30:17.244058    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0e23997_ca08_48ae_b74c_f2a97f47ca2d.slice/crio-ebd0d19ec24740e54afb2d045df22ca263b669b952eee169ae77f2fcdc7250b3 WatchSource:0}: Error finding container ebd0d19ec24740e54afb2d045df22ca263b669b952eee169ae77f2fcdc7250b3: Status 404 returned error can't find the container with id ebd0d19ec24740e54afb2d045df22ca263b669b952eee169ae77f2fcdc7250b3
Sep 15 10:30:17.841147 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:17.841114    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7891-53fdce" event={"ID":"c0e23997-ca08-48ae-b74c-f2a97f47ca2d","Type":"ContainerStarted","Data":"ebd0d19ec24740e54afb2d045df22ca263b669b952eee169ae77f2fcdc7250b3"}
Sep 15 10:30:17.866079 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:17.866046    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:30:21.853898 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:21.853864    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"]
Sep 15 10:30:21.858116 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:21.858019    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:30:21.863447 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:21.862970    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-86a8f5-dockercfg-2t4n6\""
Sep 15 10:30:21.863447 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:21.863360    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-86a8f5\""
Sep 15 10:30:21.870343 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:21.870310    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"]
Sep 15 10:30:21.876501 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:21.876469    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb\" (UID: \"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:30:21.876623 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:21.876561    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5g2\" (UniqueName: \"kubernetes.io/projected/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-kube-api-access-sl5g2\") pod \"payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb\" (UID: \"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:30:21.977606 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:21.977566    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb\" (UID: \"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:30:21.977899 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:21.977679    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5g2\" (UniqueName: \"kubernetes.io/projected/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-kube-api-access-sl5g2\") pod \"payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb\" (UID: \"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:30:21.978316 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:21.978293    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb\" (UID: \"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:30:21.986638 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:21.986614    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5g2\" (UniqueName: \"kubernetes.io/projected/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-kube-api-access-sl5g2\") pod \"payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb\" (UID: \"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223\") " pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:30:22.176985 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:22.176905    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:30:22.870862 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:22.870835    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7891-53fdce" event={"ID":"c0e23997-ca08-48ae-b74c-f2a97f47ca2d","Type":"ContainerStarted","Data":"040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73"}
Sep 15 10:30:22.885565 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:22.885519    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7891-53fdce" podStartSLOduration=1.344007814 podStartE2EDuration="6.8855062s" totalImagesPullingTime="5.541498386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:30:22.884406196 +0000 UTC m=+741.578561879" watchObservedRunningTime="2026-09-15 10:30:22.8855062 +0000 UTC m=+741.579661892"
Sep 15 10:30:22.994748 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:22.994719    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"]
Sep 15 10:30:22.997175 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:30:22.997146    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5bbc13_dd07_4bb9_94a7_7ca7b9eb2223.slice/crio-e8f26109a0413b5c1c4f45bd747713dca1a2ce1074a7750ccd9cfcddf65eb980 WatchSource:0}: Error finding container e8f26109a0413b5c1c4f45bd747713dca1a2ce1074a7750ccd9cfcddf65eb980: Status 404 returned error can't find the container with id e8f26109a0413b5c1c4f45bd747713dca1a2ce1074a7750ccd9cfcddf65eb980
Sep 15 10:30:23.465343 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.465303    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7891-53fdce"]
Sep 15 10:30:23.468913 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.468884    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7891-53fdce"]
Sep 15 10:30:23.578367 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.578288    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7891-0329b3"]
Sep 15 10:30:23.578625 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.578614    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e23997-ca08-48ae-b74c-f2a97f47ca2d" containerName="test-curl-7891-53fdce"
Sep 15 10:30:23.578667 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.578627    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e23997-ca08-48ae-b74c-f2a97f47ca2d" containerName="test-curl-7891-53fdce"
Sep 15 10:30:23.578706 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.578681    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e23997-ca08-48ae-b74c-f2a97f47ca2d" containerName="test-curl-7891-53fdce"
Sep 15 10:30:23.582058 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.582037    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-0329b3"
Sep 15 10:30:23.587533 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.587506    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7891-0329b3"]
Sep 15 10:30:23.694452 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.694414    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxjfw\" (UniqueName: \"kubernetes.io/projected/e01ff212-d81c-40ed-87d6-771c4b55edef-kube-api-access-wxjfw\") pod \"test-curl-7891-0329b3\" (UID: \"e01ff212-d81c-40ed-87d6-771c4b55edef\") " pod="opendatahub/test-curl-7891-0329b3"
Sep 15 10:30:23.795265 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.795226    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxjfw\" (UniqueName: \"kubernetes.io/projected/e01ff212-d81c-40ed-87d6-771c4b55edef-kube-api-access-wxjfw\") pod \"test-curl-7891-0329b3\" (UID: \"e01ff212-d81c-40ed-87d6-771c4b55edef\") " pod="opendatahub/test-curl-7891-0329b3"
Sep 15 10:30:23.805247 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.805212    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxjfw\" (UniqueName: \"kubernetes.io/projected/e01ff212-d81c-40ed-87d6-771c4b55edef-kube-api-access-wxjfw\") pod \"test-curl-7891-0329b3\" (UID: \"e01ff212-d81c-40ed-87d6-771c4b55edef\") " pod="opendatahub/test-curl-7891-0329b3"
Sep 15 10:30:23.878730 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.878648    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb" event={"ID":"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223","Type":"ContainerStarted","Data":"e8f26109a0413b5c1c4f45bd747713dca1a2ce1074a7750ccd9cfcddf65eb980"}
Sep 15 10:30:23.879213 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:30:23.879116    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7891-53fdce" container="test-curl-7891-53fdce"
Sep 15 10:30:23.893065 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:23.893035    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-0329b3"
Sep 15 10:30:24.178421 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:24.178339    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7891-0329b3"]
Sep 15 10:30:24.182111 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:30:24.182074    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode01ff212_d81c_40ed_87d6_771c4b55edef.slice/crio-c400675a9500a12eb13c623e3a6e524e689230dba58c1b057bba5eccd03e5cc0 WatchSource:0}: Error finding container c400675a9500a12eb13c623e3a6e524e689230dba58c1b057bba5eccd03e5cc0: Status 404 returned error can't find the container with id c400675a9500a12eb13c623e3a6e524e689230dba58c1b057bba5eccd03e5cc0
Sep 15 10:30:24.886204 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:24.886165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7891-0329b3" event={"ID":"e01ff212-d81c-40ed-87d6-771c4b55edef","Type":"ContainerStarted","Data":"22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f"}
Sep 15 10:30:24.886656 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:24.886216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7891-0329b3" event={"ID":"e01ff212-d81c-40ed-87d6-771c4b55edef","Type":"ContainerStarted","Data":"c400675a9500a12eb13c623e3a6e524e689230dba58c1b057bba5eccd03e5cc0"}
Sep 15 10:30:24.886656 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:24.886331    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7891-53fdce" podUID="c0e23997-ca08-48ae-b74c-f2a97f47ca2d" containerName="test-curl-7891-53fdce" containerID="cri-o://040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73" gracePeriod=2
Sep 15 10:30:24.927453 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:24.927399    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7891-0329b3" podStartSLOduration=1.9273821930000001 podStartE2EDuration="1.927382193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:30:24.926215826 +0000 UTC m=+743.620371514" watchObservedRunningTime="2026-09-15 10:30:24.927382193 +0000 UTC m=+743.621537865"
Sep 15 10:30:25.544596 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:25.544549    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7891-0329b3"]
Sep 15 10:30:25.549906 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:25.549873    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7891-0329b3"]
Sep 15 10:30:25.893042 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:25.893007    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb" event={"ID":"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223","Type":"ContainerStarted","Data":"8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076"}
Sep 15 10:30:25.893516 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:25.893087    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:30:25.893516 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:30:25.893245    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7891-0329b3" container="test-curl-7891-0329b3"
Sep 15 10:30:25.912195 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:25.912144    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb" podStartSLOduration=2.497511791 podStartE2EDuration="4.912123796s" totalImagesPullingTime="2.414612005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:30:25.910582038 +0000 UTC m=+744.604737722" watchObservedRunningTime="2026-09-15 10:30:25.912123796 +0000 UTC m=+744.606279479"
Sep 15 10:30:26.897906 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:26.897864    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7891-0329b3" podUID="e01ff212-d81c-40ed-87d6-771c4b55edef" containerName="test-curl-7891-0329b3" containerID="cri-o://22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f" gracePeriod=2
Sep 15 10:30:27.076420 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:27.076399    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-53fdce"
Sep 15 10:30:27.127284 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:27.127247    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4k4b2\" (UniqueName: \"kubernetes.io/projected/c0e23997-ca08-48ae-b74c-f2a97f47ca2d-kube-api-access-4k4b2\") pod \"c0e23997-ca08-48ae-b74c-f2a97f47ca2d\" (UID: \"c0e23997-ca08-48ae-b74c-f2a97f47ca2d\") "
Sep 15 10:30:27.129417 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:27.129390    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e23997-ca08-48ae-b74c-f2a97f47ca2d-kube-api-access-4k4b2" (OuterVolumeSpecName: "kube-api-access-4k4b2") pod "c0e23997-ca08-48ae-b74c-f2a97f47ca2d" (UID: "c0e23997-ca08-48ae-b74c-f2a97f47ca2d"). InnerVolumeSpecName "kube-api-access-4k4b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:30:27.228829 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:27.228726    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k4b2\" (UniqueName: \"kubernetes.io/projected/c0e23997-ca08-48ae-b74c-f2a97f47ca2d-kube-api-access-4k4b2\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:30:27.786083 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:27.786034    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0e23997-ca08-48ae-b74c-f2a97f47ca2d" path="/var/lib/kubelet/pods/c0e23997-ca08-48ae-b74c-f2a97f47ca2d/volumes"
Sep 15 10:30:27.903980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:27.903947    2584 generic.go:356] "Generic (PLEG): container finished" podID="c0e23997-ca08-48ae-b74c-f2a97f47ca2d" containerID="040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73" exitCode=137
Sep 15 10:30:27.903980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:27.903985    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-53fdce"
Sep 15 10:30:27.904438 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:27.904046    2584 scope.go:117] "RemoveContainer" containerID="040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73"
Sep 15 10:30:27.912872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:27.912853    2584 scope.go:117] "RemoveContainer" containerID="040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73"
Sep 15 10:30:27.913186 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:30:27.913166    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73\": container with ID starting with 040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73 not found: ID does not exist" containerID="040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73"
Sep 15 10:30:27.913233 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:27.913195    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73"} err="failed to get container status \"040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73\": rpc error: code = NotFound desc = could not find container \"040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73\": container with ID starting with 040d03863dac5c6bc22d59a2e2d2333d76a037070269544ea236b1d658283d73 not found: ID does not exist"
Sep 15 10:30:29.034081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:29.034059    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-0329b3"
Sep 15 10:30:29.147053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:29.146963    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxjfw\" (UniqueName: \"kubernetes.io/projected/e01ff212-d81c-40ed-87d6-771c4b55edef-kube-api-access-wxjfw\") pod \"e01ff212-d81c-40ed-87d6-771c4b55edef\" (UID: \"e01ff212-d81c-40ed-87d6-771c4b55edef\") "
Sep 15 10:30:29.149005 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:29.148984    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01ff212-d81c-40ed-87d6-771c4b55edef-kube-api-access-wxjfw" (OuterVolumeSpecName: "kube-api-access-wxjfw") pod "e01ff212-d81c-40ed-87d6-771c4b55edef" (UID: "e01ff212-d81c-40ed-87d6-771c4b55edef"). InnerVolumeSpecName "kube-api-access-wxjfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:30:29.248554 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:29.248521    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxjfw\" (UniqueName: \"kubernetes.io/projected/e01ff212-d81c-40ed-87d6-771c4b55edef-kube-api-access-wxjfw\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:30:29.785156 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:29.785120    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e01ff212-d81c-40ed-87d6-771c4b55edef" path="/var/lib/kubelet/pods/e01ff212-d81c-40ed-87d6-771c4b55edef/volumes"
Sep 15 10:30:29.915336 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:29.915306    2584 generic.go:356] "Generic (PLEG): container finished" podID="e01ff212-d81c-40ed-87d6-771c4b55edef" containerID="22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f" exitCode=137
Sep 15 10:30:29.915498 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:29.915359    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-0329b3"
Sep 15 10:30:29.915498 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:29.915402    2584 scope.go:117] "RemoveContainer" containerID="22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f"
Sep 15 10:30:29.923196 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:29.923179    2584 scope.go:117] "RemoveContainer" containerID="22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f"
Sep 15 10:30:29.923471 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:30:29.923451    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f\": container with ID starting with 22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f not found: ID does not exist" containerID="22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f"
Sep 15 10:30:29.923523 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:29.923478    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f"} err="failed to get container status \"22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f\": rpc error: code = NotFound desc = could not find container \"22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f\": container with ID starting with 22ff67acb09806ecf9c1d36cc4ee1ddf8aae7d0364a86513dd75a69848dd1d0f not found: ID does not exist"
Sep 15 10:30:30.562290 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.562255    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"]
Sep 15 10:30:30.562669 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.562600    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01ff212-d81c-40ed-87d6-771c4b55edef" containerName="test-curl-7891-0329b3"
Sep 15 10:30:30.562669 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.562611    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01ff212-d81c-40ed-87d6-771c4b55edef" containerName="test-curl-7891-0329b3"
Sep 15 10:30:30.562669 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.562667    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e01ff212-d81c-40ed-87d6-771c4b55edef" containerName="test-curl-7891-0329b3"
Sep 15 10:30:30.566866 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.566846    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:30.569132 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.569109    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 10:30:30.569238 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.569171    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 10:30:30.570016 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.569992    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b8e8c34b-serving-cert\""
Sep 15 10:30:30.570066 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.570021    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 10:30:30.570115 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.569994    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dkr42\""
Sep 15 10:30:30.578081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.578047    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"]
Sep 15 10:30:30.661021 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.660983    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-metrics-certs\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:30.661191 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.661059    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf8c5cc-c13c-48ab-80a6-276553c6745e-openshift-service-ca\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:30.661191 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.661085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52jld\" (UniqueName: \"kubernetes.io/projected/eaf8c5cc-c13c-48ab-80a6-276553c6745e-kube-api-access-52jld\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:30.661191 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.661111    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-maas-api-tls\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:30.761831 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.761759    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-metrics-certs\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:30.762030 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.761862    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf8c5cc-c13c-48ab-80a6-276553c6745e-openshift-service-ca\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:30.762030 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.761886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52jld\" (UniqueName: \"kubernetes.io/projected/eaf8c5cc-c13c-48ab-80a6-276553c6745e-kube-api-access-52jld\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:30.762030 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.761912    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-maas-api-tls\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:30.762030 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:30:30.761994    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-serving-cert: secret "maas-api-e2e-ait-b8e8c34b-serving-cert" not found
Sep 15 10:30:30.762250 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:30:30.762052    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-maas-api-tls podName:eaf8c5cc-c13c-48ab-80a6-276553c6745e nodeName:}" failed. No retries permitted until 2026-09-15 10:30:31.26203607 +0000 UTC m=+749.956191741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-maas-api-tls") pod "maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc" (UID: "eaf8c5cc-c13c-48ab-80a6-276553c6745e") : secret "maas-api-e2e-ait-b8e8c34b-serving-cert" not found
Sep 15 10:30:30.762574 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.762552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf8c5cc-c13c-48ab-80a6-276553c6745e-openshift-service-ca\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:30.764389 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.764354    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-metrics-certs\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:30.770105 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:30.770086    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52jld\" (UniqueName: \"kubernetes.io/projected/eaf8c5cc-c13c-48ab-80a6-276553c6745e-kube-api-access-52jld\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:31.267030 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.266982    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-maas-api-tls\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:31.269363 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.269329    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-maas-api-tls\") pod \"maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:31.486775 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.486732    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:31.561286 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.561257    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"]
Sep 15 10:30:31.565220 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.565198    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:30:31.567625 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.567602    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b8e8c34b\""
Sep 15 10:30:31.567738 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.567656    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b8e8c34b-dockercfg-j5vp5\""
Sep 15 10:30:31.573232 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.573204    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"]
Sep 15 10:30:31.670433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.670388    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/118d54d4-1d3e-4501-8f7b-9ce0786b9210-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw\" (UID: \"118d54d4-1d3e-4501-8f7b-9ce0786b9210\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:30:31.670613 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.670465    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvkk8\" (UniqueName: \"kubernetes.io/projected/118d54d4-1d3e-4501-8f7b-9ce0786b9210-kube-api-access-qvkk8\") pod \"payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw\" (UID: \"118d54d4-1d3e-4501-8f7b-9ce0786b9210\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:30:31.759146 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.759112    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"]
Sep 15 10:30:31.762103 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:30:31.762076    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf8c5cc_c13c_48ab_80a6_276553c6745e.slice/crio-6de156755667eefa9f5bae0c4663042de4d6f679aa7d895c4a657751c5030f7d WatchSource:0}: Error finding container 6de156755667eefa9f5bae0c4663042de4d6f679aa7d895c4a657751c5030f7d: Status 404 returned error can't find the container with id 6de156755667eefa9f5bae0c4663042de4d6f679aa7d895c4a657751c5030f7d
Sep 15 10:30:31.771137 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.771112    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/118d54d4-1d3e-4501-8f7b-9ce0786b9210-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw\" (UID: \"118d54d4-1d3e-4501-8f7b-9ce0786b9210\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:30:31.771235 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.771194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvkk8\" (UniqueName: \"kubernetes.io/projected/118d54d4-1d3e-4501-8f7b-9ce0786b9210-kube-api-access-qvkk8\") pod \"payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw\" (UID: \"118d54d4-1d3e-4501-8f7b-9ce0786b9210\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:30:31.771770 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.771752    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/118d54d4-1d3e-4501-8f7b-9ce0786b9210-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw\" (UID: \"118d54d4-1d3e-4501-8f7b-9ce0786b9210\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:30:31.780961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.780937    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvkk8\" (UniqueName: \"kubernetes.io/projected/118d54d4-1d3e-4501-8f7b-9ce0786b9210-kube-api-access-qvkk8\") pod \"payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw\" (UID: \"118d54d4-1d3e-4501-8f7b-9ce0786b9210\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:30:31.878098 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.878014    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:30:31.927110 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:31.927077    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc" event={"ID":"eaf8c5cc-c13c-48ab-80a6-276553c6745e","Type":"ContainerStarted","Data":"6de156755667eefa9f5bae0c4663042de4d6f679aa7d895c4a657751c5030f7d"}
Sep 15 10:30:32.159688 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:32.159628    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"]
Sep 15 10:30:32.162499 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:30:32.162462    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118d54d4_1d3e_4501_8f7b_9ce0786b9210.slice/crio-d46df3958acc155168e526cc0d573bb77f9c8979f4fb08bc5142f470281e7d7b WatchSource:0}: Error finding container d46df3958acc155168e526cc0d573bb77f9c8979f4fb08bc5142f470281e7d7b: Status 404 returned error can't find the container with id d46df3958acc155168e526cc0d573bb77f9c8979f4fb08bc5142f470281e7d7b
Sep 15 10:30:32.936851 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:32.936791    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw" event={"ID":"118d54d4-1d3e-4501-8f7b-9ce0786b9210","Type":"ContainerStarted","Data":"9a76560aafa4f3171d2e242b23c60a16b7a8a033f6d58ef47aa05ec8feeb43be"}
Sep 15 10:30:32.937344 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:32.936880    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw" event={"ID":"118d54d4-1d3e-4501-8f7b-9ce0786b9210","Type":"ContainerStarted","Data":"d46df3958acc155168e526cc0d573bb77f9c8979f4fb08bc5142f470281e7d7b"}
Sep 15 10:30:32.937344 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:32.936909    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:30:32.956015 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:32.955951    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw" podStartSLOduration=1.95592717 podStartE2EDuration="1.95592717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:30:32.954650325 +0000 UTC m=+751.648806010" watchObservedRunningTime="2026-09-15 10:30:32.95592717 +0000 UTC m=+751.650082850"
Sep 15 10:30:34.955980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:34.955944    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc" event={"ID":"eaf8c5cc-c13c-48ab-80a6-276553c6745e","Type":"ContainerStarted","Data":"42e0ec2f3a2ae16b8bb8a98c51d9c0edd597ed3e6822baec2e5f8d5c3c410ade"}
Sep 15 10:30:34.956362 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:34.956081    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:34.974503 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:34.974447    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc" podStartSLOduration=2.790380015 podStartE2EDuration="4.974428363s" totalImagesPullingTime="2.184048348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:30:34.973385772 +0000 UTC m=+753.667541465" watchObservedRunningTime="2026-09-15 10:30:34.974428363 +0000 UTC m=+753.668584043"
Sep 15 10:30:35.215522 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.215439    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"]
Sep 15 10:30:35.218915 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.218897    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:30:35.221479 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.221446    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-8110bc-dockercfg-2gsjl\""
Sep 15 10:30:35.221598 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.221552    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-8110bc\""
Sep 15 10:30:35.227511 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.227487    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"]
Sep 15 10:30:35.235319 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.235295    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"]
Sep 15 10:30:35.238907 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.238882    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:30:35.250501 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.250476    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"]
Sep 15 10:30:35.314481 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.314448    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2\" (UID: \"c9c7a156-ebf1-4fa4-ac73-3e6201da3239\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:30:35.314676 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.314518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dwqh\" (UniqueName: \"kubernetes.io/projected/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-kube-api-access-9dwqh\") pod \"payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2\" (UID: \"c9c7a156-ebf1-4fa4-ac73-3e6201da3239\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:30:35.314676 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.314596    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336409e-2a24-4b0e-863f-c496baeacf99-plugins-config-volume\") pod \"payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj\" (UID: \"f336409e-2a24-4b0e-863f-c496baeacf99\") " pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:30:35.314676 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.314628    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7v4s\" (UniqueName: \"kubernetes.io/projected/f336409e-2a24-4b0e-863f-c496baeacf99-kube-api-access-t7v4s\") pod \"payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj\" (UID: \"f336409e-2a24-4b0e-863f-c496baeacf99\") " pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:30:35.415790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.415751    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336409e-2a24-4b0e-863f-c496baeacf99-plugins-config-volume\") pod \"payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj\" (UID: \"f336409e-2a24-4b0e-863f-c496baeacf99\") " pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:30:35.415790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.415809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7v4s\" (UniqueName: \"kubernetes.io/projected/f336409e-2a24-4b0e-863f-c496baeacf99-kube-api-access-t7v4s\") pod \"payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj\" (UID: \"f336409e-2a24-4b0e-863f-c496baeacf99\") " pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:30:35.416046 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.415859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2\" (UID: \"c9c7a156-ebf1-4fa4-ac73-3e6201da3239\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:30:35.416046 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.415952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dwqh\" (UniqueName: \"kubernetes.io/projected/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-kube-api-access-9dwqh\") pod \"payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2\" (UID: \"c9c7a156-ebf1-4fa4-ac73-3e6201da3239\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:30:35.416501 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.416479    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336409e-2a24-4b0e-863f-c496baeacf99-plugins-config-volume\") pod \"payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj\" (UID: \"f336409e-2a24-4b0e-863f-c496baeacf99\") " pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:30:35.416641 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.416621    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2\" (UID: \"c9c7a156-ebf1-4fa4-ac73-3e6201da3239\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:30:35.423553 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.423527    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dwqh\" (UniqueName: \"kubernetes.io/projected/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-kube-api-access-9dwqh\") pod \"payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2\" (UID: \"c9c7a156-ebf1-4fa4-ac73-3e6201da3239\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:30:35.423711 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.423693    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7v4s\" (UniqueName: \"kubernetes.io/projected/f336409e-2a24-4b0e-863f-c496baeacf99-kube-api-access-t7v4s\") pod \"payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj\" (UID: \"f336409e-2a24-4b0e-863f-c496baeacf99\") " pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:30:35.530154 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.530122    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:30:35.552321 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.552291    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:30:35.804768 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.804743    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"]
Sep 15 10:30:35.806622 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:30:35.806596    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf336409e_2a24_4b0e_863f_c496baeacf99.slice/crio-21bef3ae81a05d526f7a424f30c39600de22f815913286a62a7631fdaa0856fa WatchSource:0}: Error finding container 21bef3ae81a05d526f7a424f30c39600de22f815913286a62a7631fdaa0856fa: Status 404 returned error can't find the container with id 21bef3ae81a05d526f7a424f30c39600de22f815913286a62a7631fdaa0856fa
Sep 15 10:30:35.829904 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.829874    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"]
Sep 15 10:30:35.835866 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:30:35.835840    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c7a156_ebf1_4fa4_ac73_3e6201da3239.slice/crio-1a995718ad465a102666848032c3931a5c95dd77532e5081dcb15433a3e2a1fc WatchSource:0}: Error finding container 1a995718ad465a102666848032c3931a5c95dd77532e5081dcb15433a3e2a1fc: Status 404 returned error can't find the container with id 1a995718ad465a102666848032c3931a5c95dd77532e5081dcb15433a3e2a1fc
Sep 15 10:30:35.962423 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.962379    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2" event={"ID":"c9c7a156-ebf1-4fa4-ac73-3e6201da3239","Type":"ContainerStarted","Data":"ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8"}
Sep 15 10:30:35.962906 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.962539    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:30:35.962906 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.962563    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2" event={"ID":"c9c7a156-ebf1-4fa4-ac73-3e6201da3239","Type":"ContainerStarted","Data":"1a995718ad465a102666848032c3931a5c95dd77532e5081dcb15433a3e2a1fc"}
Sep 15 10:30:35.964135 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.964112    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj" event={"ID":"f336409e-2a24-4b0e-863f-c496baeacf99","Type":"ContainerStarted","Data":"62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861"}
Sep 15 10:30:35.964249 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.964152    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj" event={"ID":"f336409e-2a24-4b0e-863f-c496baeacf99","Type":"ContainerStarted","Data":"21bef3ae81a05d526f7a424f30c39600de22f815913286a62a7631fdaa0856fa"}
Sep 15 10:30:35.980274 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.980220    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2" podStartSLOduration=0.980203775 podStartE2EDuration="980.203775ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:30:35.978072324 +0000 UTC m=+754.672228005" watchObservedRunningTime="2026-09-15 10:30:35.980203775 +0000 UTC m=+754.674359504"
Sep 15 10:30:35.996238 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:35.996167    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj" podStartSLOduration=0.996146782 podStartE2EDuration="996.146782ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:30:35.99473999 +0000 UTC m=+754.688895671" watchObservedRunningTime="2026-09-15 10:30:35.996146782 +0000 UTC m=+754.690302464"
Sep 15 10:30:36.155595 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:36.155503    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:30:36.899636 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:36.899599    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:30:36.970198 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:36.970161    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:30:40.589504 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.589465    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"]
Sep 15 10:30:40.593152 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.593132    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:30:40.595333 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.595312    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-234840\""
Sep 15 10:30:40.595436 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.595339    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-234840-dockercfg-lx9rd\""
Sep 15 10:30:40.606922 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.606896    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"]
Sep 15 10:30:40.712392 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.712362    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"]
Sep 15 10:30:40.715717 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.715697    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:30:40.729621 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.729598    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"]
Sep 15 10:30:40.770038 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.770007    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8swn\" (UniqueName: \"kubernetes.io/projected/e4c0f1f5-1dd4-4878-9a90-c97579113319-kube-api-access-r8swn\") pod \"payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv\" (UID: \"e4c0f1f5-1dd4-4878-9a90-c97579113319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:30:40.770189 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.770088    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c0f1f5-1dd4-4878-9a90-c97579113319-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv\" (UID: \"e4c0f1f5-1dd4-4878-9a90-c97579113319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:30:40.871489 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.871415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8swn\" (UniqueName: \"kubernetes.io/projected/e4c0f1f5-1dd4-4878-9a90-c97579113319-kube-api-access-r8swn\") pod \"payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv\" (UID: \"e4c0f1f5-1dd4-4878-9a90-c97579113319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:30:40.871489 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.871450    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ae53bd-2579-40a0-8100-ac32d8f2d600-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf\" (UID: \"38ae53bd-2579-40a0-8100-ac32d8f2d600\") " pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:30:40.871706 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.871492    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c0f1f5-1dd4-4878-9a90-c97579113319-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv\" (UID: \"e4c0f1f5-1dd4-4878-9a90-c97579113319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:30:40.871706 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.871553    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgmjc\" (UniqueName: \"kubernetes.io/projected/38ae53bd-2579-40a0-8100-ac32d8f2d600-kube-api-access-jgmjc\") pod \"payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf\" (UID: \"38ae53bd-2579-40a0-8100-ac32d8f2d600\") " pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:30:40.872243 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.872218    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c0f1f5-1dd4-4878-9a90-c97579113319-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv\" (UID: \"e4c0f1f5-1dd4-4878-9a90-c97579113319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:30:40.879553 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.879528    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8swn\" (UniqueName: \"kubernetes.io/projected/e4c0f1f5-1dd4-4878-9a90-c97579113319-kube-api-access-r8swn\") pod \"payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv\" (UID: \"e4c0f1f5-1dd4-4878-9a90-c97579113319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:30:40.903857 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.903832    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:30:40.971587 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.971564    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:40.972139 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.972111    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ae53bd-2579-40a0-8100-ac32d8f2d600-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf\" (UID: \"38ae53bd-2579-40a0-8100-ac32d8f2d600\") " pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:30:40.972271 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.972207    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgmjc\" (UniqueName: \"kubernetes.io/projected/38ae53bd-2579-40a0-8100-ac32d8f2d600-kube-api-access-jgmjc\") pod \"payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf\" (UID: \"38ae53bd-2579-40a0-8100-ac32d8f2d600\") " pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:30:40.972774 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.972741    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ae53bd-2579-40a0-8100-ac32d8f2d600-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf\" (UID: \"38ae53bd-2579-40a0-8100-ac32d8f2d600\") " pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:30:40.981264 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:40.981174    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgmjc\" (UniqueName: \"kubernetes.io/projected/38ae53bd-2579-40a0-8100-ac32d8f2d600-kube-api-access-jgmjc\") pod \"payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf\" (UID: \"38ae53bd-2579-40a0-8100-ac32d8f2d600\") " pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:30:41.025964 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:41.025931    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:30:41.173441 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:41.173413    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"]
Sep 15 10:30:41.176099 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:30:41.176071    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c0f1f5_1dd4_4878_9a90_c97579113319.slice/crio-68234d197502d0f8d31d6a9ddf421ad062e516fa668ab8e1a74c59be97ef7f28 WatchSource:0}: Error finding container 68234d197502d0f8d31d6a9ddf421ad062e516fa668ab8e1a74c59be97ef7f28: Status 404 returned error can't find the container with id 68234d197502d0f8d31d6a9ddf421ad062e516fa668ab8e1a74c59be97ef7f28
Sep 15 10:30:41.305051 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:41.305021    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"]
Sep 15 10:30:41.306944 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:30:41.306915    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ae53bd_2579_40a0_8100_ac32d8f2d600.slice/crio-66be11792ad528838575edf88426987b1db163f97439d9b895df3ccfc33358a3 WatchSource:0}: Error finding container 66be11792ad528838575edf88426987b1db163f97439d9b895df3ccfc33358a3: Status 404 returned error can't find the container with id 66be11792ad528838575edf88426987b1db163f97439d9b895df3ccfc33358a3
Sep 15 10:30:42.003013 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:42.002972    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf" event={"ID":"38ae53bd-2579-40a0-8100-ac32d8f2d600","Type":"ContainerStarted","Data":"8786ea15d7b61a934fadcd7169713ffb5b99c1f07a29533279bd50fcae84973f"}
Sep 15 10:30:42.003467 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:42.003068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf" event={"ID":"38ae53bd-2579-40a0-8100-ac32d8f2d600","Type":"ContainerStarted","Data":"66be11792ad528838575edf88426987b1db163f97439d9b895df3ccfc33358a3"}
Sep 15 10:30:42.003467 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:42.003101    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:30:42.004659 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:42.004637    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv" event={"ID":"e4c0f1f5-1dd4-4878-9a90-c97579113319","Type":"ContainerStarted","Data":"83c8e3e9b3b45ecf2add8a7f430776fb45b0df38b4d6aabf6616836f94c57045"}
Sep 15 10:30:42.004783 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:42.004671    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv" event={"ID":"e4c0f1f5-1dd4-4878-9a90-c97579113319","Type":"ContainerStarted","Data":"68234d197502d0f8d31d6a9ddf421ad062e516fa668ab8e1a74c59be97ef7f28"}
Sep 15 10:30:42.004783 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:42.004691    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:30:42.018981 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:42.018939    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf" podStartSLOduration=2.018926852 podStartE2EDuration="2.018926852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:30:42.017997639 +0000 UTC m=+760.712153371" watchObservedRunningTime="2026-09-15 10:30:42.018926852 +0000 UTC m=+760.713082595"
Sep 15 10:30:42.034224 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:42.034181    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv" podStartSLOduration=2.034165816 podStartE2EDuration="2.034165816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:30:42.033093421 +0000 UTC m=+760.727249103" watchObservedRunningTime="2026-09-15 10:30:42.034165816 +0000 UTC m=+760.728321493"
Sep 15 10:30:43.950536 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:43.950501    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:30:46.972380 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:46.972350    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:30:47.976287 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:47.976261    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:30:52.045697 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.045661    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"]
Sep 15 10:30:52.052079 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.052055    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.055105 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.055081    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8c1047\"/\"simulator-w0-8c1047-kserve-self-signed-certs\""
Sep 15 10:30:52.055238 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.055085    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8c1047\"/\"kube-root-ca.crt\""
Sep 15 10:30:52.055775 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.055750    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8c1047\"/\"openshift-service-ca.crt\""
Sep 15 10:30:52.056668 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.056641    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8c1047\"/\"default-dockercfg-ff5nk\""
Sep 15 10:30:52.058430 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.058406    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"]
Sep 15 10:30:52.070814 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.070780    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-home\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.070921 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.070852    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-dshm\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.070921 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.070880    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7ds\" (UniqueName: \"kubernetes.io/projected/699599fa-ce28-4c30-983d-778af87e42f4-kube-api-access-vn7ds\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.070921 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.070906    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-model-cache\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.071040 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.070941    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-tmp-dir\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.071040 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.070965    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699599fa-ce28-4c30-983d-778af87e42f4-tls-certs\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.171601 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.171566    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-home\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.171752 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.171623    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-dshm\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.171843 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.171749    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7ds\" (UniqueName: \"kubernetes.io/projected/699599fa-ce28-4c30-983d-778af87e42f4-kube-api-access-vn7ds\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.171843 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.171823    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-model-cache\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.171935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.171853    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-tmp-dir\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.171935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.171879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699599fa-ce28-4c30-983d-778af87e42f4-tls-certs\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.172127 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.172104    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-home\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.172292 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.172271    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-model-cache\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.172380 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.172362    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-tmp-dir\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.173886 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.173865    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-dshm\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.174364 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.174344    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699599fa-ce28-4c30-983d-778af87e42f4-tls-certs\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.183747 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.183724    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7ds\" (UniqueName: \"kubernetes.io/projected/699599fa-ce28-4c30-983d-778af87e42f4-kube-api-access-vn7ds\") pod \"simulator-w0-8c1047-kserve-86c59487bb-9kf8c\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") " pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.369960 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.369870    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:30:52.641910 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:52.641833    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"]
Sep 15 10:30:52.644863 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:30:52.644838    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod699599fa_ce28_4c30_983d_778af87e42f4.slice/crio-901a99ee18fb8d7d57fddfb6cf7a6e560605567fbf76086f6d9f9355250a4d9c WatchSource:0}: Error finding container 901a99ee18fb8d7d57fddfb6cf7a6e560605567fbf76086f6d9f9355250a4d9c: Status 404 returned error can't find the container with id 901a99ee18fb8d7d57fddfb6cf7a6e560605567fbf76086f6d9f9355250a4d9c
Sep 15 10:30:53.011010 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:53.010978    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:30:53.011242 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:53.011145    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:30:53.061790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:53.061757    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c" event={"ID":"699599fa-ce28-4c30-983d-778af87e42f4","Type":"ContainerStarted","Data":"9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd"}
Sep 15 10:30:53.062285 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:53.061817    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c" event={"ID":"699599fa-ce28-4c30-983d-778af87e42f4","Type":"ContainerStarted","Data":"901a99ee18fb8d7d57fddfb6cf7a6e560605567fbf76086f6d9f9355250a4d9c"}
Sep 15 10:30:53.079500 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:53.079444    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c" podStartSLOduration=1.079428331 podStartE2EDuration="1.079428331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:30:53.077516915 +0000 UTC m=+771.771672595" watchObservedRunningTime="2026-09-15 10:30:53.079428331 +0000 UTC m=+771.773584057"
Sep 15 10:30:58.798881 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:58.798841    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"]
Sep 15 10:30:58.799354 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:58.799091    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc" podUID="eaf8c5cc-c13c-48ab-80a6-276553c6745e" containerName="maas-api" containerID="cri-o://42e0ec2f3a2ae16b8bb8a98c51d9c0edd597ed3e6822baec2e5f8d5c3c410ade" gracePeriod=30
Sep 15 10:30:58.855906 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:58.855876    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"]
Sep 15 10:30:58.856150 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:58.856123    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw" podUID="118d54d4-1d3e-4501-8f7b-9ce0786b9210" containerName="payload-pre-processing" containerID="cri-o://9a76560aafa4f3171d2e242b23c60a16b7a8a033f6d58ef47aa05ec8feeb43be" gracePeriod=30
Sep 15 10:30:59.094320 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.094118    2584 generic.go:356] "Generic (PLEG): container finished" podID="eaf8c5cc-c13c-48ab-80a6-276553c6745e" containerID="42e0ec2f3a2ae16b8bb8a98c51d9c0edd597ed3e6822baec2e5f8d5c3c410ade" exitCode=0
Sep 15 10:30:59.094320 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.094215    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc" event={"ID":"eaf8c5cc-c13c-48ab-80a6-276553c6745e","Type":"ContainerDied","Data":"42e0ec2f3a2ae16b8bb8a98c51d9c0edd597ed3e6822baec2e5f8d5c3c410ade"}
Sep 15 10:30:59.096683 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.096654    2584 generic.go:356] "Generic (PLEG): container finished" podID="118d54d4-1d3e-4501-8f7b-9ce0786b9210" containerID="9a76560aafa4f3171d2e242b23c60a16b7a8a033f6d58ef47aa05ec8feeb43be" exitCode=0
Sep 15 10:30:59.096839 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.096739    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw" event={"ID":"118d54d4-1d3e-4501-8f7b-9ce0786b9210","Type":"ContainerDied","Data":"9a76560aafa4f3171d2e242b23c60a16b7a8a033f6d58ef47aa05ec8feeb43be"}
Sep 15 10:30:59.134641 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.134611    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:30:59.154335 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.154306    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:30:59.181249 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.181216    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:30:59.231756 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.231719    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52jld\" (UniqueName: \"kubernetes.io/projected/eaf8c5cc-c13c-48ab-80a6-276553c6745e-kube-api-access-52jld\") pod \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") "
Sep 15 10:30:59.231964 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.231782    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-metrics-certs\") pod \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") "
Sep 15 10:30:59.231964 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.231821    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-maas-api-tls\") pod \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") "
Sep 15 10:30:59.231964 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.231867    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf8c5cc-c13c-48ab-80a6-276553c6745e-openshift-service-ca\") pod \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\" (UID: \"eaf8c5cc-c13c-48ab-80a6-276553c6745e\") "
Sep 15 10:30:59.231964 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.231897    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/118d54d4-1d3e-4501-8f7b-9ce0786b9210-plugins-config-volume\") pod \"118d54d4-1d3e-4501-8f7b-9ce0786b9210\" (UID: \"118d54d4-1d3e-4501-8f7b-9ce0786b9210\") "
Sep 15 10:30:59.231964 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.231938    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvkk8\" (UniqueName: \"kubernetes.io/projected/118d54d4-1d3e-4501-8f7b-9ce0786b9210-kube-api-access-qvkk8\") pod \"118d54d4-1d3e-4501-8f7b-9ce0786b9210\" (UID: \"118d54d4-1d3e-4501-8f7b-9ce0786b9210\") "
Sep 15 10:30:59.232754 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.232439    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf8c5cc-c13c-48ab-80a6-276553c6745e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eaf8c5cc-c13c-48ab-80a6-276553c6745e" (UID: "eaf8c5cc-c13c-48ab-80a6-276553c6745e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:30:59.232754 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.232741    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118d54d4-1d3e-4501-8f7b-9ce0786b9210-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "118d54d4-1d3e-4501-8f7b-9ce0786b9210" (UID: "118d54d4-1d3e-4501-8f7b-9ce0786b9210"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:30:59.235459 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.235001    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf8c5cc-c13c-48ab-80a6-276553c6745e-kube-api-access-52jld" (OuterVolumeSpecName: "kube-api-access-52jld") pod "eaf8c5cc-c13c-48ab-80a6-276553c6745e" (UID: "eaf8c5cc-c13c-48ab-80a6-276553c6745e"). InnerVolumeSpecName "kube-api-access-52jld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:30:59.235459 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.235124    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eaf8c5cc-c13c-48ab-80a6-276553c6745e" (UID: "eaf8c5cc-c13c-48ab-80a6-276553c6745e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:30:59.235632 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.235482    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eaf8c5cc-c13c-48ab-80a6-276553c6745e" (UID: "eaf8c5cc-c13c-48ab-80a6-276553c6745e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:30:59.236830 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.236779    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118d54d4-1d3e-4501-8f7b-9ce0786b9210-kube-api-access-qvkk8" (OuterVolumeSpecName: "kube-api-access-qvkk8") pod "118d54d4-1d3e-4501-8f7b-9ce0786b9210" (UID: "118d54d4-1d3e-4501-8f7b-9ce0786b9210"). InnerVolumeSpecName "kube-api-access-qvkk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:30:59.333823 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.333720    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52jld\" (UniqueName: \"kubernetes.io/projected/eaf8c5cc-c13c-48ab-80a6-276553c6745e-kube-api-access-52jld\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:30:59.333823 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.333752    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-metrics-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:30:59.333823 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.333761    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf8c5cc-c13c-48ab-80a6-276553c6745e-maas-api-tls\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:30:59.333823 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.333769    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf8c5cc-c13c-48ab-80a6-276553c6745e-openshift-service-ca\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:30:59.333823 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.333777    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/118d54d4-1d3e-4501-8f7b-9ce0786b9210-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:30:59.333823 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:30:59.333785    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvkk8\" (UniqueName: \"kubernetes.io/projected/118d54d4-1d3e-4501-8f7b-9ce0786b9210-kube-api-access-qvkk8\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:31:00.103276 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:00.103195    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"
Sep 15 10:31:00.103851 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:00.103195    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc" event={"ID":"eaf8c5cc-c13c-48ab-80a6-276553c6745e","Type":"ContainerDied","Data":"6de156755667eefa9f5bae0c4663042de4d6f679aa7d895c4a657751c5030f7d"}
Sep 15 10:31:00.103851 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:00.103327    2584 scope.go:117] "RemoveContainer" containerID="42e0ec2f3a2ae16b8bb8a98c51d9c0edd597ed3e6822baec2e5f8d5c3c410ade"
Sep 15 10:31:00.104717 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:00.104576    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw" event={"ID":"118d54d4-1d3e-4501-8f7b-9ce0786b9210","Type":"ContainerDied","Data":"d46df3958acc155168e526cc0d573bb77f9c8979f4fb08bc5142f470281e7d7b"}
Sep 15 10:31:00.104717 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:00.104656    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"
Sep 15 10:31:00.112164 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:00.112147    2584 scope.go:117] "RemoveContainer" containerID="9a76560aafa4f3171d2e242b23c60a16b7a8a033f6d58ef47aa05ec8feeb43be"
Sep 15 10:31:00.142605 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:00.142573    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"]
Sep 15 10:31:00.165896 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:00.165862    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b8e8c34b-ff8fffccd-2bcbc"]
Sep 15 10:31:00.182121 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:00.182090    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"]
Sep 15 10:31:00.186327 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:00.186300    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b8e8c34b-567ff9ccb6-wrmqw"]
Sep 15 10:31:01.785675 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:01.785643    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="118d54d4-1d3e-4501-8f7b-9ce0786b9210" path="/var/lib/kubelet/pods/118d54d4-1d3e-4501-8f7b-9ce0786b9210/volumes"
Sep 15 10:31:01.786063 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:01.786049    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaf8c5cc-c13c-48ab-80a6-276553c6745e" path="/var/lib/kubelet/pods/eaf8c5cc-c13c-48ab-80a6-276553c6745e/volumes"
Sep 15 10:31:02.370111 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:02.370079    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:31:02.370365 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:02.370343    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:31:02.382765 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:02.382741    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:31:03.131883 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:03.131850    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:31:06.161309 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.161272    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"]
Sep 15 10:31:06.161720 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.161640    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf8c5cc-c13c-48ab-80a6-276553c6745e" containerName="maas-api"
Sep 15 10:31:06.161720 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.161650    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf8c5cc-c13c-48ab-80a6-276553c6745e" containerName="maas-api"
Sep 15 10:31:06.161720 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.161662    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118d54d4-1d3e-4501-8f7b-9ce0786b9210" containerName="payload-pre-processing"
Sep 15 10:31:06.161720 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.161667    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="118d54d4-1d3e-4501-8f7b-9ce0786b9210" containerName="payload-pre-processing"
Sep 15 10:31:06.161898 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.161723    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="118d54d4-1d3e-4501-8f7b-9ce0786b9210" containerName="payload-pre-processing"
Sep 15 10:31:06.161898 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.161732    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaf8c5cc-c13c-48ab-80a6-276553c6745e" containerName="maas-api"
Sep 15 10:31:06.165576 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.165554    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:06.167841 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.167794    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b797d492-dockercfg-skgw7\""
Sep 15 10:31:06.167841 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.167832    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b797d492\""
Sep 15 10:31:06.173839 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.173817    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"]
Sep 15 10:31:06.207719 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.207690    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c005b208-e646-4053-80c3-3de6944b05a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd\" (UID: \"c005b208-e646-4053-80c3-3de6944b05a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:06.207719 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.207720    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxtcp\" (UniqueName: \"kubernetes.io/projected/c005b208-e646-4053-80c3-3de6944b05a5-kube-api-access-wxtcp\") pod \"payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd\" (UID: \"c005b208-e646-4053-80c3-3de6944b05a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:06.308743 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.308700    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c005b208-e646-4053-80c3-3de6944b05a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd\" (UID: \"c005b208-e646-4053-80c3-3de6944b05a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:06.308743 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.308743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxtcp\" (UniqueName: \"kubernetes.io/projected/c005b208-e646-4053-80c3-3de6944b05a5-kube-api-access-wxtcp\") pod \"payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd\" (UID: \"c005b208-e646-4053-80c3-3de6944b05a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:06.309408 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.309390    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c005b208-e646-4053-80c3-3de6944b05a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd\" (UID: \"c005b208-e646-4053-80c3-3de6944b05a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:06.316918 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.316898    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxtcp\" (UniqueName: \"kubernetes.io/projected/c005b208-e646-4053-80c3-3de6944b05a5-kube-api-access-wxtcp\") pod \"payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd\" (UID: \"c005b208-e646-4053-80c3-3de6944b05a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:06.476680 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.476599    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:06.749237 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:06.749214    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"]
Sep 15 10:31:06.752006 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:31:06.751975    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc005b208_e646_4053_80c3_3de6944b05a5.slice/crio-b29e6c6da21af1914f52b924888bae983fecedfaab9474898266dae2099b0859 WatchSource:0}: Error finding container b29e6c6da21af1914f52b924888bae983fecedfaab9474898266dae2099b0859: Status 404 returned error can't find the container with id b29e6c6da21af1914f52b924888bae983fecedfaab9474898266dae2099b0859
Sep 15 10:31:07.142509 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:07.142419    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd" event={"ID":"c005b208-e646-4053-80c3-3de6944b05a5","Type":"ContainerStarted","Data":"bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2"}
Sep 15 10:31:07.142666 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:07.142563    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:07.142666 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:07.142620    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd" event={"ID":"c005b208-e646-4053-80c3-3de6944b05a5","Type":"ContainerStarted","Data":"b29e6c6da21af1914f52b924888bae983fecedfaab9474898266dae2099b0859"}
Sep 15 10:31:07.159877 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:07.159829    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd" podStartSLOduration=1.159815528 podStartE2EDuration="1.159815528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:31:07.157958144 +0000 UTC m=+785.852113825" watchObservedRunningTime="2026-09-15 10:31:07.159815528 +0000 UTC m=+785.853971505"
Sep 15 10:31:18.148589 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:18.148557    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:20.741616 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.741577    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"]
Sep 15 10:31:20.745438 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.745421    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.749252 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.749225    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ff0655\"/\"default-dockercfg-vvpdr\""
Sep 15 10:31:20.749383 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.749229    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ff0655\"/\"simulator-w4-ff0655-kserve-self-signed-certs\""
Sep 15 10:31:20.749383 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.749288    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ff0655\"/\"openshift-service-ca.crt\""
Sep 15 10:31:20.749383 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.749253    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ff0655\"/\"kube-root-ca.crt\""
Sep 15 10:31:20.755025 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.755003    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"]
Sep 15 10:31:20.850387 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.850345    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-model-cache\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.850387 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.850387    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-tmp-dir\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.850674 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.850538    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-home\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.850674 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.850583    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpzw6\" (UniqueName: \"kubernetes.io/projected/bb3153b5-85b6-4565-b625-b9ab7092c37d-kube-api-access-xpzw6\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.850674 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.850638    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-dshm\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.850674 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.850666    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb3153b5-85b6-4565-b625-b9ab7092c37d-tls-certs\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.951735 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.951704    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-home\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.951965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.951740    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzw6\" (UniqueName: \"kubernetes.io/projected/bb3153b5-85b6-4565-b625-b9ab7092c37d-kube-api-access-xpzw6\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.951965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.951767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-dshm\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.951965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.951850    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb3153b5-85b6-4565-b625-b9ab7092c37d-tls-certs\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.951965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.951931    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-model-cache\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.951965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.951964    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-tmp-dir\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.952258 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.952194    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-home\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.952258 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.952242    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-model-cache\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.952343 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.952300    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-tmp-dir\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.954067 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.954045    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-dshm\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.954226 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.954206    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb3153b5-85b6-4565-b625-b9ab7092c37d-tls-certs\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:20.959415 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:20.959396    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzw6\" (UniqueName: \"kubernetes.io/projected/bb3153b5-85b6-4565-b625-b9ab7092c37d-kube-api-access-xpzw6\") pod \"simulator-w4-ff0655-kserve-5554db4bbf-jjbbw\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") " pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:21.056090 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:21.055992    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:21.323990 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:21.323925    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"]
Sep 15 10:31:21.326283 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:31:21.326255    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb3153b5_85b6_4565_b625_b9ab7092c37d.slice/crio-16793abe951177d0fff6eb6c1fb5b6dae3e761644660d176499d195e685447f4 WatchSource:0}: Error finding container 16793abe951177d0fff6eb6c1fb5b6dae3e761644660d176499d195e685447f4: Status 404 returned error can't find the container with id 16793abe951177d0fff6eb6c1fb5b6dae3e761644660d176499d195e685447f4
Sep 15 10:31:22.214155 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:22.214118    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw" event={"ID":"bb3153b5-85b6-4565-b625-b9ab7092c37d","Type":"ContainerStarted","Data":"d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e"}
Sep 15 10:31:22.214155 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:22.214165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw" event={"ID":"bb3153b5-85b6-4565-b625-b9ab7092c37d","Type":"ContainerStarted","Data":"16793abe951177d0fff6eb6c1fb5b6dae3e761644660d176499d195e685447f4"}
Sep 15 10:31:22.234491 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:22.234450    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw" podStartSLOduration=2.234436136 podStartE2EDuration="2.234436136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:31:22.2334341 +0000 UTC m=+800.927589783" watchObservedRunningTime="2026-09-15 10:31:22.234436136 +0000 UTC m=+800.928591816"
Sep 15 10:31:25.786160 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:25.786073    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:31:27.629150 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:27.629106    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"]
Sep 15 10:31:27.632462 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:27.632447    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"
Sep 15 10:31:27.635079 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:27.635059    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 10:31:27.635195 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:27.635061    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jrzm\""
Sep 15 10:31:27.639514 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:27.639490    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"]
Sep 15 10:31:27.714503 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:27.714455    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870524fe-a5f7-482b-b93e-783bfee64590-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b797d492-7lr78\" (UID: \"870524fe-a5f7-482b-b93e-783bfee64590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"
Sep 15 10:31:27.815575 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:27.815533    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870524fe-a5f7-482b-b93e-783bfee64590-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b797d492-7lr78\" (UID: \"870524fe-a5f7-482b-b93e-783bfee64590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"
Sep 15 10:31:27.816355 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:27.816327    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870524fe-a5f7-482b-b93e-783bfee64590-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b797d492-7lr78\" (UID: \"870524fe-a5f7-482b-b93e-783bfee64590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"
Sep 15 10:31:27.945738 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:27.945661    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"
Sep 15 10:31:28.214164 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:28.214139    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"]
Sep 15 10:31:28.216749 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:31:28.216723    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870524fe_a5f7_482b_b93e_783bfee64590.slice/crio-8550f7d79049b3a1406489725585518f1e9474689b42fcc5f1dcd2f2c460d19d WatchSource:0}: Error finding container 8550f7d79049b3a1406489725585518f1e9474689b42fcc5f1dcd2f2c460d19d: Status 404 returned error can't find the container with id 8550f7d79049b3a1406489725585518f1e9474689b42fcc5f1dcd2f2c460d19d
Sep 15 10:31:28.242385 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:28.242357    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78" event={"ID":"870524fe-a5f7-482b-b93e-783bfee64590","Type":"ContainerStarted","Data":"8550f7d79049b3a1406489725585518f1e9474689b42fcc5f1dcd2f2c460d19d"}
Sep 15 10:31:31.056550 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:31.056520    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:31.056949 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:31.056562    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:31.068669 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:31.068646    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:31.259108 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:31.259033    2584 generic.go:356] "Generic (PLEG): container finished" podID="870524fe-a5f7-482b-b93e-783bfee64590" containerID="f9bfa384ce9666e99c98d58352eb2de47aaba9002e68a84bcb0a6bb433aaf569" exitCode=0
Sep 15 10:31:31.259108 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:31.259078    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78" event={"ID":"870524fe-a5f7-482b-b93e-783bfee64590","Type":"ContainerDied","Data":"f9bfa384ce9666e99c98d58352eb2de47aaba9002e68a84bcb0a6bb433aaf569"}
Sep 15 10:31:31.270378 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:31.270358    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:31:32.401006 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:32.400984    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"
Sep 15 10:31:32.452724 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:32.452694    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870524fe-a5f7-482b-b93e-783bfee64590-maas-api-service-ca\") pod \"870524fe-a5f7-482b-b93e-783bfee64590\" (UID: \"870524fe-a5f7-482b-b93e-783bfee64590\") "
Sep 15 10:31:32.453073 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:32.453053    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870524fe-a5f7-482b-b93e-783bfee64590-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "870524fe-a5f7-482b-b93e-783bfee64590" (UID: "870524fe-a5f7-482b-b93e-783bfee64590"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:31:32.554072 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:32.553988    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870524fe-a5f7-482b-b93e-783bfee64590-maas-api-service-ca\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:31:33.269936 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:33.269897    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78" event={"ID":"870524fe-a5f7-482b-b93e-783bfee64590","Type":"ContainerDied","Data":"8550f7d79049b3a1406489725585518f1e9474689b42fcc5f1dcd2f2c460d19d"}
Sep 15 10:31:33.269936 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:33.269927    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"
Sep 15 10:31:33.270157 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:33.269958    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8550f7d79049b3a1406489725585518f1e9474689b42fcc5f1dcd2f2c460d19d"
Sep 15 10:31:34.560499 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:34.560466    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"]
Sep 15 10:31:34.561290 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:34.561228    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd" podUID="c005b208-e646-4053-80c3-3de6944b05a5" containerName="payload-pre-processing" containerID="cri-o://bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2" gracePeriod=30
Sep 15 10:31:34.704675 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:34.704641    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"]
Sep 15 10:31:34.709101 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:34.709071    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b797d492-7lr78"]
Sep 15 10:31:34.834517 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:34.834493    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:34.876722 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:34.876696    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c005b208-e646-4053-80c3-3de6944b05a5-plugins-config-volume\") pod \"c005b208-e646-4053-80c3-3de6944b05a5\" (UID: \"c005b208-e646-4053-80c3-3de6944b05a5\") "
Sep 15 10:31:34.876899 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:34.876726    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxtcp\" (UniqueName: \"kubernetes.io/projected/c005b208-e646-4053-80c3-3de6944b05a5-kube-api-access-wxtcp\") pod \"c005b208-e646-4053-80c3-3de6944b05a5\" (UID: \"c005b208-e646-4053-80c3-3de6944b05a5\") "
Sep 15 10:31:34.877140 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:34.877109    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c005b208-e646-4053-80c3-3de6944b05a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c005b208-e646-4053-80c3-3de6944b05a5" (UID: "c005b208-e646-4053-80c3-3de6944b05a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:31:34.878718 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:34.878693    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c005b208-e646-4053-80c3-3de6944b05a5-kube-api-access-wxtcp" (OuterVolumeSpecName: "kube-api-access-wxtcp") pod "c005b208-e646-4053-80c3-3de6944b05a5" (UID: "c005b208-e646-4053-80c3-3de6944b05a5"). InnerVolumeSpecName "kube-api-access-wxtcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:31:34.977879 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:34.977851    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c005b208-e646-4053-80c3-3de6944b05a5-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:31:34.977879 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:34.977876    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxtcp\" (UniqueName: \"kubernetes.io/projected/c005b208-e646-4053-80c3-3de6944b05a5-kube-api-access-wxtcp\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:31:35.280015 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:35.279985    2584 generic.go:356] "Generic (PLEG): container finished" podID="c005b208-e646-4053-80c3-3de6944b05a5" containerID="bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2" exitCode=0
Sep 15 10:31:35.280189 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:35.280023    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"
Sep 15 10:31:35.280189 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:35.280055    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd" event={"ID":"c005b208-e646-4053-80c3-3de6944b05a5","Type":"ContainerDied","Data":"bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2"}
Sep 15 10:31:35.280189 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:35.280106    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd" event={"ID":"c005b208-e646-4053-80c3-3de6944b05a5","Type":"ContainerDied","Data":"b29e6c6da21af1914f52b924888bae983fecedfaab9474898266dae2099b0859"}
Sep 15 10:31:35.280189 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:35.280129    2584 scope.go:117] "RemoveContainer" containerID="bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2"
Sep 15 10:31:35.288756 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:35.288737    2584 scope.go:117] "RemoveContainer" containerID="bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2"
Sep 15 10:31:35.289013 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:31:35.288994    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2\": container with ID starting with bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2 not found: ID does not exist" containerID="bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2"
Sep 15 10:31:35.289081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:35.289025    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2"} err="failed to get container status \"bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2\": rpc error: code = NotFound desc = could not find container \"bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2\": container with ID starting with bab8d8ae85ffa8c428666350c4b08c766de1bb5263d1ff680d2b04381f8019a2 not found: ID does not exist"
Sep 15 10:31:35.300615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:35.300588    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"]
Sep 15 10:31:35.304705 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:35.304681    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b797d492-5ffd6b8458-8kvhd"]
Sep 15 10:31:35.786647 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:35.786611    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="870524fe-a5f7-482b-b93e-783bfee64590" path="/var/lib/kubelet/pods/870524fe-a5f7-482b-b93e-783bfee64590/volumes"
Sep 15 10:31:35.787236 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:35.787136    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c005b208-e646-4053-80c3-3de6944b05a5" path="/var/lib/kubelet/pods/c005b208-e646-4053-80c3-3de6944b05a5/volumes"
Sep 15 10:31:39.559743 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.559702    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"]
Sep 15 10:31:39.560238 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.560208    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="870524fe-a5f7-482b-b93e-783bfee64590" containerName="revoke-keys"
Sep 15 10:31:39.560238 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.560222    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="870524fe-a5f7-482b-b93e-783bfee64590" containerName="revoke-keys"
Sep 15 10:31:39.560238 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.560234    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c005b208-e646-4053-80c3-3de6944b05a5" containerName="payload-pre-processing"
Sep 15 10:31:39.560238 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.560240    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c005b208-e646-4053-80c3-3de6944b05a5" containerName="payload-pre-processing"
Sep 15 10:31:39.560488 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.560302    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="870524fe-a5f7-482b-b93e-783bfee64590" containerName="revoke-keys"
Sep 15 10:31:39.560488 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.560313    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c005b208-e646-4053-80c3-3de6944b05a5" containerName="payload-pre-processing"
Sep 15 10:31:39.564696 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.564659    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.567452 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.567423    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dkr42\""
Sep 15 10:31:39.567570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.567493    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 10:31:39.567570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.567511    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 10:31:39.567747 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.567502    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 10:31:39.567823 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.567745    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-68c3a296-serving-cert\""
Sep 15 10:31:39.572026 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.571922    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"]
Sep 15 10:31:39.618407 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.618372    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-openshift-service-ca\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.618562 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.618415    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-maas-api-tls\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.618562 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.618437    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-metrics-certs\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.618562 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.618515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcnrc\" (UniqueName: \"kubernetes.io/projected/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-kube-api-access-xcnrc\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.719251 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.719215    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-openshift-service-ca\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.719459 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.719266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-maas-api-tls\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.719459 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.719290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-metrics-certs\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.719459 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.719357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcnrc\" (UniqueName: \"kubernetes.io/projected/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-kube-api-access-xcnrc\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.720016 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.719991    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-openshift-service-ca\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.721794 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.721765    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-metrics-certs\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.721994 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.721970    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-maas-api-tls\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.727325 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.727304    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcnrc\" (UniqueName: \"kubernetes.io/projected/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-kube-api-access-xcnrc\") pod \"maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:39.877300 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:39.877215    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:40.152633 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.152475    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"]
Sep 15 10:31:40.155095 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:31:40.155065    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b11bd8a_351f_426f_8aee_d8df6db1b4cf.slice/crio-1e5a0885440c41c3965329baf800eac15f8c291687f70a40d841fdc269cd1e09 WatchSource:0}: Error finding container 1e5a0885440c41c3965329baf800eac15f8c291687f70a40d841fdc269cd1e09: Status 404 returned error can't find the container with id 1e5a0885440c41c3965329baf800eac15f8c291687f70a40d841fdc269cd1e09
Sep 15 10:31:40.287792 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.287757    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"]
Sep 15 10:31:40.292402 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.292381    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:31:40.294741 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.294716    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-68c3a296\""
Sep 15 10:31:40.294897 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.294871    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-68c3a296-dockercfg-f8tzq\""
Sep 15 10:31:40.302419 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.302390    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"]
Sep 15 10:31:40.306399 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.306363    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg" event={"ID":"1b11bd8a-351f-426f-8aee-d8df6db1b4cf","Type":"ContainerStarted","Data":"1e5a0885440c41c3965329baf800eac15f8c291687f70a40d841fdc269cd1e09"}
Sep 15 10:31:40.324293 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.324262    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-plugins-config-volume\") pod \"payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf\" (UID: \"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b\") " pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:31:40.324372 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.324341    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsdjs\" (UniqueName: \"kubernetes.io/projected/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-kube-api-access-xsdjs\") pod \"payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf\" (UID: \"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b\") " pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:31:40.425231 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.425163    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-plugins-config-volume\") pod \"payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf\" (UID: \"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b\") " pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:31:40.425231 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.425207    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsdjs\" (UniqueName: \"kubernetes.io/projected/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-kube-api-access-xsdjs\") pod \"payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf\" (UID: \"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b\") " pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:31:40.425860 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.425841    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-plugins-config-volume\") pod \"payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf\" (UID: \"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b\") " pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:31:40.433262 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.433243    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsdjs\" (UniqueName: \"kubernetes.io/projected/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-kube-api-access-xsdjs\") pod \"payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf\" (UID: \"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b\") " pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:31:40.604463 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.604442    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:31:40.878812 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:40.878724    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"]
Sep 15 10:31:40.881321 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:31:40.881288    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7aa2dff_669e_4a26_9f3c_208e7fd05d6b.slice/crio-162d05dfdcdea75e961cf51a7ac887f2cfceb3a782db77a46056792b36c42e91 WatchSource:0}: Error finding container 162d05dfdcdea75e961cf51a7ac887f2cfceb3a782db77a46056792b36c42e91: Status 404 returned error can't find the container with id 162d05dfdcdea75e961cf51a7ac887f2cfceb3a782db77a46056792b36c42e91
Sep 15 10:31:41.314257 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:41.314214    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg" event={"ID":"1b11bd8a-351f-426f-8aee-d8df6db1b4cf","Type":"ContainerStarted","Data":"291a35447bc08422d3953e566ad449b76cb2dbae7987243cc5ff1af2781a1efc"}
Sep 15 10:31:41.314475 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:41.314270    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:41.316060 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:41.316028    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf" event={"ID":"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b","Type":"ContainerStarted","Data":"a467e90d0135e2fe7b5b727813058b3109c6ec0b18e32fe5e4b3226f5b1118c9"}
Sep 15 10:31:41.316162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:41.316082    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf" event={"ID":"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b","Type":"ContainerStarted","Data":"162d05dfdcdea75e961cf51a7ac887f2cfceb3a782db77a46056792b36c42e91"}
Sep 15 10:31:41.316162 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:41.316111    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:31:41.333133 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:41.333073    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg" podStartSLOduration=1.9405489660000002 podStartE2EDuration="2.333050036s" totalImagesPullingTime="392.50107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:31:41.331488874 +0000 UTC m=+820.025644552" watchObservedRunningTime="2026-09-15 10:31:41.333050036 +0000 UTC m=+820.027205717"
Sep 15 10:31:41.347068 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:41.347017    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf" podStartSLOduration=1.347001678 podStartE2EDuration="1.347001678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:31:41.345627514 +0000 UTC m=+820.039783196" watchObservedRunningTime="2026-09-15 10:31:41.347001678 +0000 UTC m=+820.041157384"
Sep 15 10:31:47.327107 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:47.327071    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:31:51.257119 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:51.257083    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:31:52.323352 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:31:52.323323    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:32:09.268973 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.268941    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"]
Sep 15 10:32:09.269467 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.269205    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg" podUID="1b11bd8a-351f-426f-8aee-d8df6db1b4cf" containerName="maas-api" containerID="cri-o://291a35447bc08422d3953e566ad449b76cb2dbae7987243cc5ff1af2781a1efc" gracePeriod=30
Sep 15 10:32:09.321669 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.321633    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"]
Sep 15 10:32:09.321957 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.321925    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf" podUID="f7aa2dff-669e-4a26-9f3c-208e7fd05d6b" containerName="payload-processing" containerID="cri-o://a467e90d0135e2fe7b5b727813058b3109c6ec0b18e32fe5e4b3226f5b1118c9" gracePeriod=30
Sep 15 10:32:09.450430 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.450306    2584 generic.go:356] "Generic (PLEG): container finished" podID="1b11bd8a-351f-426f-8aee-d8df6db1b4cf" containerID="291a35447bc08422d3953e566ad449b76cb2dbae7987243cc5ff1af2781a1efc" exitCode=0
Sep 15 10:32:09.450430 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.450384    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg" event={"ID":"1b11bd8a-351f-426f-8aee-d8df6db1b4cf","Type":"ContainerDied","Data":"291a35447bc08422d3953e566ad449b76cb2dbae7987243cc5ff1af2781a1efc"}
Sep 15 10:32:09.452138 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.452110    2584 generic.go:356] "Generic (PLEG): container finished" podID="f7aa2dff-669e-4a26-9f3c-208e7fd05d6b" containerID="a467e90d0135e2fe7b5b727813058b3109c6ec0b18e32fe5e4b3226f5b1118c9" exitCode=0
Sep 15 10:32:09.452250 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.452170    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf" event={"ID":"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b","Type":"ContainerDied","Data":"a467e90d0135e2fe7b5b727813058b3109c6ec0b18e32fe5e4b3226f5b1118c9"}
Sep 15 10:32:09.571191 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.571165    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:32:09.596310 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.596286    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:32:09.707691 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.707647    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-plugins-config-volume\") pod \"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b\" (UID: \"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b\") "
Sep 15 10:32:09.707962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.707718    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-openshift-service-ca\") pod \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") "
Sep 15 10:32:09.707962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.707844    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-maas-api-tls\") pod \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") "
Sep 15 10:32:09.707962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.707889    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-metrics-certs\") pod \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") "
Sep 15 10:32:09.707962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.707934    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsdjs\" (UniqueName: \"kubernetes.io/projected/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-kube-api-access-xsdjs\") pod \"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b\" (UID: \"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b\") "
Sep 15 10:32:09.708198 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.707988    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcnrc\" (UniqueName: \"kubernetes.io/projected/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-kube-api-access-xcnrc\") pod \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\" (UID: \"1b11bd8a-351f-426f-8aee-d8df6db1b4cf\") "
Sep 15 10:32:09.708198 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.708079    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b11bd8a-351f-426f-8aee-d8df6db1b4cf" (UID: "1b11bd8a-351f-426f-8aee-d8df6db1b4cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:32:09.708198 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.708126    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7aa2dff-669e-4a26-9f3c-208e7fd05d6b" (UID: "f7aa2dff-669e-4a26-9f3c-208e7fd05d6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:32:09.708352 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.708311    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:32:09.708352 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.708331    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-openshift-service-ca\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:32:09.709980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.709953    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b11bd8a-351f-426f-8aee-d8df6db1b4cf" (UID: "1b11bd8a-351f-426f-8aee-d8df6db1b4cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:32:09.710187 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.710165    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-kube-api-access-xcnrc" (OuterVolumeSpecName: "kube-api-access-xcnrc") pod "1b11bd8a-351f-426f-8aee-d8df6db1b4cf" (UID: "1b11bd8a-351f-426f-8aee-d8df6db1b4cf"). InnerVolumeSpecName "kube-api-access-xcnrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:32:09.710465 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.710439    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1b11bd8a-351f-426f-8aee-d8df6db1b4cf" (UID: "1b11bd8a-351f-426f-8aee-d8df6db1b4cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:32:09.710552 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.710469    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-kube-api-access-xsdjs" (OuterVolumeSpecName: "kube-api-access-xsdjs") pod "f7aa2dff-669e-4a26-9f3c-208e7fd05d6b" (UID: "f7aa2dff-669e-4a26-9f3c-208e7fd05d6b"). InnerVolumeSpecName "kube-api-access-xsdjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:32:09.808915 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.808889    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsdjs\" (UniqueName: \"kubernetes.io/projected/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b-kube-api-access-xsdjs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:32:09.808915 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.808914    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcnrc\" (UniqueName: \"kubernetes.io/projected/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-kube-api-access-xcnrc\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:32:09.809056 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.808923    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-maas-api-tls\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:32:09.809056 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:09.808932    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b11bd8a-351f-426f-8aee-d8df6db1b4cf-metrics-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:32:10.465423 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:10.465381    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"
Sep 15 10:32:10.465959 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:10.465386    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf" event={"ID":"f7aa2dff-669e-4a26-9f3c-208e7fd05d6b","Type":"ContainerDied","Data":"162d05dfdcdea75e961cf51a7ac887f2cfceb3a782db77a46056792b36c42e91"}
Sep 15 10:32:10.465959 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:10.465566    2584 scope.go:117] "RemoveContainer" containerID="a467e90d0135e2fe7b5b727813058b3109c6ec0b18e32fe5e4b3226f5b1118c9"
Sep 15 10:32:10.467362 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:10.467340    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg" event={"ID":"1b11bd8a-351f-426f-8aee-d8df6db1b4cf","Type":"ContainerDied","Data":"1e5a0885440c41c3965329baf800eac15f8c291687f70a40d841fdc269cd1e09"}
Sep 15 10:32:10.467473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:10.467441    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"
Sep 15 10:32:10.475148 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:10.475132    2584 scope.go:117] "RemoveContainer" containerID="291a35447bc08422d3953e566ad449b76cb2dbae7987243cc5ff1af2781a1efc"
Sep 15 10:32:10.485121 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:10.485093    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"]
Sep 15 10:32:10.488983 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:10.488962    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68c3a296-5bc9fc57c5-8d6kf"]
Sep 15 10:32:10.502438 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:10.502406    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"]
Sep 15 10:32:10.505532 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:10.505506    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68c3a296-7f8bf9dcdc-qwwlg"]
Sep 15 10:32:11.785619 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:11.785586    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b11bd8a-351f-426f-8aee-d8df6db1b4cf" path="/var/lib/kubelet/pods/1b11bd8a-351f-426f-8aee-d8df6db1b4cf/volumes"
Sep 15 10:32:11.786010 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:11.785977    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7aa2dff-669e-4a26-9f3c-208e7fd05d6b" path="/var/lib/kubelet/pods/f7aa2dff-669e-4a26-9f3c-208e7fd05d6b/volumes"
Sep 15 10:32:17.556485 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:17.556449    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:32:18.087684 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.087646    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"]
Sep 15 10:32:18.088055 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.088041    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b11bd8a-351f-426f-8aee-d8df6db1b4cf" containerName="maas-api"
Sep 15 10:32:18.088055 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.088056    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b11bd8a-351f-426f-8aee-d8df6db1b4cf" containerName="maas-api"
Sep 15 10:32:18.088171 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.088064    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7aa2dff-669e-4a26-9f3c-208e7fd05d6b" containerName="payload-processing"
Sep 15 10:32:18.088171 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.088070    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7aa2dff-669e-4a26-9f3c-208e7fd05d6b" containerName="payload-processing"
Sep 15 10:32:18.088171 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.088134    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7aa2dff-669e-4a26-9f3c-208e7fd05d6b" containerName="payload-processing"
Sep 15 10:32:18.088171 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.088142    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b11bd8a-351f-426f-8aee-d8df6db1b4cf" containerName="maas-api"
Sep 15 10:32:18.092588 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.092569    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.095748 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.095727    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 10:32:18.095873 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.095725    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 10:32:18.095873 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.095725    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 10:32:18.096001 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.095725    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-7b3aacf5-serving-cert\""
Sep 15 10:32:18.096001 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.095742    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dkr42\""
Sep 15 10:32:18.100685 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.100649    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"]
Sep 15 10:32:18.185006 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.184962    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdmpr\" (UniqueName: \"kubernetes.io/projected/3732c3e1-54b5-4736-b01c-2086a755ccce-kube-api-access-rdmpr\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.185217 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.185122    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-maas-api-tls\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.185217 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.185182    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3732c3e1-54b5-4736-b01c-2086a755ccce-openshift-service-ca\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.185217 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.185212    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-metrics-certs\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.286210 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.286170    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3732c3e1-54b5-4736-b01c-2086a755ccce-openshift-service-ca\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.286210 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.286213    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-metrics-certs\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.286456 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.286283    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdmpr\" (UniqueName: \"kubernetes.io/projected/3732c3e1-54b5-4736-b01c-2086a755ccce-kube-api-access-rdmpr\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.286456 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.286303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-maas-api-tls\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.286880 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.286857    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3732c3e1-54b5-4736-b01c-2086a755ccce-openshift-service-ca\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.288651 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.288627    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-metrics-certs\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.288815 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.288783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-maas-api-tls\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.293361 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.293336    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdmpr\" (UniqueName: \"kubernetes.io/projected/3732c3e1-54b5-4736-b01c-2086a755ccce-kube-api-access-rdmpr\") pod \"maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.404370 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.404271    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:18.690587 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:18.690497    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"]
Sep 15 10:32:18.694862 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:32:18.694832    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3732c3e1_54b5_4736_b01c_2086a755ccce.slice/crio-31772db8692bf9e302bdb33fda6ab1d43b504b9f201454cdf5a96615c0ea8028 WatchSource:0}: Error finding container 31772db8692bf9e302bdb33fda6ab1d43b504b9f201454cdf5a96615c0ea8028: Status 404 returned error can't find the container with id 31772db8692bf9e302bdb33fda6ab1d43b504b9f201454cdf5a96615c0ea8028
Sep 15 10:32:19.514370 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:19.514340    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz" event={"ID":"3732c3e1-54b5-4736-b01c-2086a755ccce","Type":"ContainerStarted","Data":"84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79"}
Sep 15 10:32:19.514536 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:19.514390    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz" event={"ID":"3732c3e1-54b5-4736-b01c-2086a755ccce","Type":"ContainerStarted","Data":"31772db8692bf9e302bdb33fda6ab1d43b504b9f201454cdf5a96615c0ea8028"}
Sep 15 10:32:19.514536 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:19.514420    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:19.529639 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:19.529596    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz" podStartSLOduration=1.094191314 podStartE2EDuration="1.529580839s" totalImagesPullingTime="435.389525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:32:19.528783807 +0000 UTC m=+858.222939502" watchObservedRunningTime="2026-09-15 10:32:19.529580839 +0000 UTC m=+858.223736558"
Sep 15 10:32:25.529568 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:25.529540    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:37.654501 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:37.654463    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:32:56.232318 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.232214    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"]
Sep 15 10:32:56.232939 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.232469    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz" podUID="3732c3e1-54b5-4736-b01c-2086a755ccce" containerName="maas-api" containerID="cri-o://84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79" gracePeriod=30
Sep 15 10:32:56.481405 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.481381    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:56.537376 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.537341    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-maas-api-tls\") pod \"3732c3e1-54b5-4736-b01c-2086a755ccce\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") "
Sep 15 10:32:56.537558 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.537391    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdmpr\" (UniqueName: \"kubernetes.io/projected/3732c3e1-54b5-4736-b01c-2086a755ccce-kube-api-access-rdmpr\") pod \"3732c3e1-54b5-4736-b01c-2086a755ccce\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") "
Sep 15 10:32:56.537558 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.537452    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3732c3e1-54b5-4736-b01c-2086a755ccce-openshift-service-ca\") pod \"3732c3e1-54b5-4736-b01c-2086a755ccce\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") "
Sep 15 10:32:56.537693 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.537583    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-metrics-certs\") pod \"3732c3e1-54b5-4736-b01c-2086a755ccce\" (UID: \"3732c3e1-54b5-4736-b01c-2086a755ccce\") "
Sep 15 10:32:56.538161 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.538133    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3732c3e1-54b5-4736-b01c-2086a755ccce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3732c3e1-54b5-4736-b01c-2086a755ccce" (UID: "3732c3e1-54b5-4736-b01c-2086a755ccce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:32:56.541112 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.541083    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3732c3e1-54b5-4736-b01c-2086a755ccce" (UID: "3732c3e1-54b5-4736-b01c-2086a755ccce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:32:56.542319 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.542286    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3732c3e1-54b5-4736-b01c-2086a755ccce-kube-api-access-rdmpr" (OuterVolumeSpecName: "kube-api-access-rdmpr") pod "3732c3e1-54b5-4736-b01c-2086a755ccce" (UID: "3732c3e1-54b5-4736-b01c-2086a755ccce"). InnerVolumeSpecName "kube-api-access-rdmpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:32:56.542404 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.542332    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3732c3e1-54b5-4736-b01c-2086a755ccce" (UID: "3732c3e1-54b5-4736-b01c-2086a755ccce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:32:56.638339 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.638303    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-maas-api-tls\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:32:56.638339 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.638336    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdmpr\" (UniqueName: \"kubernetes.io/projected/3732c3e1-54b5-4736-b01c-2086a755ccce-kube-api-access-rdmpr\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:32:56.638339 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.638347    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3732c3e1-54b5-4736-b01c-2086a755ccce-openshift-service-ca\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:32:56.638612 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.638355    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3732c3e1-54b5-4736-b01c-2086a755ccce-metrics-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:32:56.707893 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.707858    2584 generic.go:356] "Generic (PLEG): container finished" podID="3732c3e1-54b5-4736-b01c-2086a755ccce" containerID="84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79" exitCode=0
Sep 15 10:32:56.708036 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.707903    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"
Sep 15 10:32:56.708036 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.707912    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz" event={"ID":"3732c3e1-54b5-4736-b01c-2086a755ccce","Type":"ContainerDied","Data":"84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79"}
Sep 15 10:32:56.708036 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.707954    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz" event={"ID":"3732c3e1-54b5-4736-b01c-2086a755ccce","Type":"ContainerDied","Data":"31772db8692bf9e302bdb33fda6ab1d43b504b9f201454cdf5a96615c0ea8028"}
Sep 15 10:32:56.708036 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.707975    2584 scope.go:117] "RemoveContainer" containerID="84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79"
Sep 15 10:32:56.717705 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.717685    2584 scope.go:117] "RemoveContainer" containerID="84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79"
Sep 15 10:32:56.718012 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:32:56.717993    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79\": container with ID starting with 84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79 not found: ID does not exist" containerID="84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79"
Sep 15 10:32:56.718071 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.718021    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79"} err="failed to get container status \"84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79\": rpc error: code = NotFound desc = could not find container \"84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79\": container with ID starting with 84deb82ef281e98d586ecf6aeb758e707a73cf69932873648abb1246d931ef79 not found: ID does not exist"
Sep 15 10:32:56.729250 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.729227    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"]
Sep 15 10:32:56.732780 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:56.732758    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7b3aacf5-79cf965954-bkpfz"]
Sep 15 10:32:57.784964 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:32:57.784931    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3732c3e1-54b5-4736-b01c-2086a755ccce" path="/var/lib/kubelet/pods/3732c3e1-54b5-4736-b01c-2086a755ccce/volumes"
Sep 15 10:33:00.339879 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.339843    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"]
Sep 15 10:33:00.340257 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.340200    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3732c3e1-54b5-4736-b01c-2086a755ccce" containerName="maas-api"
Sep 15 10:33:00.340257 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.340213    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3732c3e1-54b5-4736-b01c-2086a755ccce" containerName="maas-api"
Sep 15 10:33:00.340348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.340282    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3732c3e1-54b5-4736-b01c-2086a755ccce" containerName="maas-api"
Sep 15 10:33:00.343263 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.343244    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.346299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.346278    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8110bc\"/\"kube-root-ca.crt\""
Sep 15 10:33:00.346531 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.346516    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8110bc\"/\"openshift-service-ca.crt\""
Sep 15 10:33:00.346531 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.346522    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8110bc\"/\"autores-model-8110bc-kserve-self-signed-certs\""
Sep 15 10:33:00.346706 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.346523    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8110bc\"/\"default-dockercfg-jcqg9\""
Sep 15 10:33:00.352813 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.352778    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"]
Sep 15 10:33:00.365739 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.365719    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-dshm\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.365864 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.365781    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-tmp-dir\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.365864 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.365823    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wwtv\" (UniqueName: \"kubernetes.io/projected/982f2f7e-9c16-429c-9481-744cd4cb712a-kube-api-access-9wwtv\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.365864 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.365854    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982f2f7e-9c16-429c-9481-744cd4cb712a-tls-certs\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.365974 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.365889    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-home\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.365974 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.365917    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-model-cache\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.466479 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.466448    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-tmp-dir\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.466479 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.466482    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wwtv\" (UniqueName: \"kubernetes.io/projected/982f2f7e-9c16-429c-9481-744cd4cb712a-kube-api-access-9wwtv\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.466701 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.466514    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982f2f7e-9c16-429c-9481-744cd4cb712a-tls-certs\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.466788 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.466766    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-home\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.466888 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.466788    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-tmp-dir\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.466888 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.466846    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-model-cache\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.467053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.466918    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-dshm\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.467053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.467044    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-home\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.467122 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.467101    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-model-cache\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.468983 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.468955    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-dshm\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.469114 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.469096    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982f2f7e-9c16-429c-9481-744cd4cb712a-tls-certs\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.473728 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.473705    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wwtv\" (UniqueName: \"kubernetes.io/projected/982f2f7e-9c16-429c-9481-744cd4cb712a-kube-api-access-9wwtv\") pod \"autores-model-8110bc-kserve-7f6dfbf6c9-gnch9\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") " pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.653108 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.653026    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:00.924362 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:00.924278    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"]
Sep 15 10:33:00.928385 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:33:00.928353    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod982f2f7e_9c16_429c_9481_744cd4cb712a.slice/crio-9c144faef36b90e00255dbf25dc64174db5d3f1c4cdb7611ae17bf5b44c2a8be WatchSource:0}: Error finding container 9c144faef36b90e00255dbf25dc64174db5d3f1c4cdb7611ae17bf5b44c2a8be: Status 404 returned error can't find the container with id 9c144faef36b90e00255dbf25dc64174db5d3f1c4cdb7611ae17bf5b44c2a8be
Sep 15 10:33:01.735763 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:01.735724    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9" event={"ID":"982f2f7e-9c16-429c-9481-744cd4cb712a","Type":"ContainerStarted","Data":"510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e"}
Sep 15 10:33:01.736176 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:01.735774    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9" event={"ID":"982f2f7e-9c16-429c-9481-744cd4cb712a","Type":"ContainerStarted","Data":"9c144faef36b90e00255dbf25dc64174db5d3f1c4cdb7611ae17bf5b44c2a8be"}
Sep 15 10:33:01.753863 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:01.753787    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9" podStartSLOduration=1.753764876 podStartE2EDuration="1.753764876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:33:01.75220298 +0000 UTC m=+900.446358661" watchObservedRunningTime="2026-09-15 10:33:01.753764876 +0000 UTC m=+900.447920558"
Sep 15 10:33:03.454036 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:03.453996    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:33:10.653363 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:10.653325    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:10.653757 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:10.653380    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:10.665785 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:10.665759    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:10.789153 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:10.789126    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:33:22.056870 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:22.056833    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:33:25.669081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.669048    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"]
Sep 15 10:33:25.672659 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.672637    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.675071 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.675046    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-86a8f5\"/\"kube-root-ca.crt\""
Sep 15 10:33:25.676107 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.676080    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-86a8f5\"/\"test-model-86a8f5-kserve-self-signed-certs\""
Sep 15 10:33:25.676107 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.676104    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-86a8f5\"/\"openshift-service-ca.crt\""
Sep 15 10:33:25.676293 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.676172    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-86a8f5\"/\"default-dockercfg-6sf72\""
Sep 15 10:33:25.680765 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.680744    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"]
Sep 15 10:33:25.806379 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.806346    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-tmp-dir\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.806379 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.806385    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-model-cache\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.806659 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.806411    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrc5r\" (UniqueName: \"kubernetes.io/projected/09d85a8a-137a-48d1-9610-2066ddad122a-kube-api-access-lrc5r\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.806659 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.806552    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d85a8a-137a-48d1-9610-2066ddad122a-tls-certs\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.806659 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.806626    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-home\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.806794 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.806662    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-dshm\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.907786 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.907741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-home\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.907987 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.907816    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-dshm\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.907987 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.907856    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-tmp-dir\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.907987 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.907888    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-model-cache\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.907987 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.907932    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrc5r\" (UniqueName: \"kubernetes.io/projected/09d85a8a-137a-48d1-9610-2066ddad122a-kube-api-access-lrc5r\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.908207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.908020    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d85a8a-137a-48d1-9610-2066ddad122a-tls-certs\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.908280 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.908252    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-home\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.908352 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.908290    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-tmp-dir\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.908410 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.908354    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-model-cache\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.910078 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.910053    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-dshm\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.910465 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.910447    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d85a8a-137a-48d1-9610-2066ddad122a-tls-certs\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.919429 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.919368    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrc5r\" (UniqueName: \"kubernetes.io/projected/09d85a8a-137a-48d1-9610-2066ddad122a-kube-api-access-lrc5r\") pod \"test-model-86a8f5-kserve-77bd97df8f-t4xsf\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") " pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:25.983379 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:25.983307    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:26.254212 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:26.254162    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"]
Sep 15 10:33:26.256906 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:33:26.256877    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09d85a8a_137a_48d1_9610_2066ddad122a.slice/crio-e92b51c0e8c071e902cd399ea70693388fe8d660c79b6691c9a7fa62735bfa39 WatchSource:0}: Error finding container e92b51c0e8c071e902cd399ea70693388fe8d660c79b6691c9a7fa62735bfa39: Status 404 returned error can't find the container with id e92b51c0e8c071e902cd399ea70693388fe8d660c79b6691c9a7fa62735bfa39
Sep 15 10:33:26.853435 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:26.853402    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf" event={"ID":"09d85a8a-137a-48d1-9610-2066ddad122a","Type":"ContainerStarted","Data":"ca5e615ca01c87e6c636c81d053473c52078e7349a611f248e5702bf51d5d534"}
Sep 15 10:33:26.853906 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:26.853446    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf" event={"ID":"09d85a8a-137a-48d1-9610-2066ddad122a","Type":"ContainerStarted","Data":"e92b51c0e8c071e902cd399ea70693388fe8d660c79b6691c9a7fa62735bfa39"}
Sep 15 10:33:26.870091 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:26.870040    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf" podStartSLOduration=1.8700247989999998 podStartE2EDuration="1.870024799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:33:26.868885967 +0000 UTC m=+925.563041651" watchObservedRunningTime="2026-09-15 10:33:26.870024799 +0000 UTC m=+925.564180479"
Sep 15 10:33:34.464385 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:34.464338    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:33:35.983769 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:35.983729    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:35.984188 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:35.983858    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:35.997521 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:35.997494    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:36.912573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:36.912540    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:33:44.469865 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.469821    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"]
Sep 15 10:33:44.470366 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.470049    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj" podUID="f336409e-2a24-4b0e-863f-c496baeacf99" containerName="payload-processing" containerID="cri-o://62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861" gracePeriod=30
Sep 15 10:33:44.477468 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.477344    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"]
Sep 15 10:33:44.477728 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.477556    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2" podUID="c9c7a156-ebf1-4fa4-ac73-3e6201da3239" containerName="payload-pre-processing" containerID="cri-o://ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8" gracePeriod=30
Sep 15 10:33:44.747111 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.747084    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:33:44.750579 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.750549    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:33:44.781059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.780839    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7v4s\" (UniqueName: \"kubernetes.io/projected/f336409e-2a24-4b0e-863f-c496baeacf99-kube-api-access-t7v4s\") pod \"f336409e-2a24-4b0e-863f-c496baeacf99\" (UID: \"f336409e-2a24-4b0e-863f-c496baeacf99\") "
Sep 15 10:33:44.781331 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.781248    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-plugins-config-volume\") pod \"c9c7a156-ebf1-4fa4-ac73-3e6201da3239\" (UID: \"c9c7a156-ebf1-4fa4-ac73-3e6201da3239\") "
Sep 15 10:33:44.781331 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.781310    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dwqh\" (UniqueName: \"kubernetes.io/projected/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-kube-api-access-9dwqh\") pod \"c9c7a156-ebf1-4fa4-ac73-3e6201da3239\" (UID: \"c9c7a156-ebf1-4fa4-ac73-3e6201da3239\") "
Sep 15 10:33:44.781517 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.781501    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336409e-2a24-4b0e-863f-c496baeacf99-plugins-config-volume\") pod \"f336409e-2a24-4b0e-863f-c496baeacf99\" (UID: \"f336409e-2a24-4b0e-863f-c496baeacf99\") "
Sep 15 10:33:44.782204 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.781727    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9c7a156-ebf1-4fa4-ac73-3e6201da3239" (UID: "c9c7a156-ebf1-4fa4-ac73-3e6201da3239"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:33:44.782204 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.781926    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:33:44.782414 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.782246    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f336409e-2a24-4b0e-863f-c496baeacf99-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f336409e-2a24-4b0e-863f-c496baeacf99" (UID: "f336409e-2a24-4b0e-863f-c496baeacf99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:33:44.783730 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.783699    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f336409e-2a24-4b0e-863f-c496baeacf99-kube-api-access-t7v4s" (OuterVolumeSpecName: "kube-api-access-t7v4s") pod "f336409e-2a24-4b0e-863f-c496baeacf99" (UID: "f336409e-2a24-4b0e-863f-c496baeacf99"). InnerVolumeSpecName "kube-api-access-t7v4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:33:44.784286 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.784254    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-kube-api-access-9dwqh" (OuterVolumeSpecName: "kube-api-access-9dwqh") pod "c9c7a156-ebf1-4fa4-ac73-3e6201da3239" (UID: "c9c7a156-ebf1-4fa4-ac73-3e6201da3239"). InnerVolumeSpecName "kube-api-access-9dwqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:33:44.883047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.883006    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dwqh\" (UniqueName: \"kubernetes.io/projected/c9c7a156-ebf1-4fa4-ac73-3e6201da3239-kube-api-access-9dwqh\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:33:44.883047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.883047    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336409e-2a24-4b0e-863f-c496baeacf99-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:33:44.883047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.883063    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7v4s\" (UniqueName: \"kubernetes.io/projected/f336409e-2a24-4b0e-863f-c496baeacf99-kube-api-access-t7v4s\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:33:44.950571 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.950535    2584 generic.go:356] "Generic (PLEG): container finished" podID="f336409e-2a24-4b0e-863f-c496baeacf99" containerID="62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861" exitCode=0
Sep 15 10:33:44.950777 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.950582    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"
Sep 15 10:33:44.950777 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.950620    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj" event={"ID":"f336409e-2a24-4b0e-863f-c496baeacf99","Type":"ContainerDied","Data":"62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861"}
Sep 15 10:33:44.950777 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.950706    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj" event={"ID":"f336409e-2a24-4b0e-863f-c496baeacf99","Type":"ContainerDied","Data":"21bef3ae81a05d526f7a424f30c39600de22f815913286a62a7631fdaa0856fa"}
Sep 15 10:33:44.950777 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.950738    2584 scope.go:117] "RemoveContainer" containerID="62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861"
Sep 15 10:33:44.952034 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.952009    2584 generic.go:356] "Generic (PLEG): container finished" podID="c9c7a156-ebf1-4fa4-ac73-3e6201da3239" containerID="ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8" exitCode=0
Sep 15 10:33:44.952119 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.952056    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"
Sep 15 10:33:44.952163 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.952131    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2" event={"ID":"c9c7a156-ebf1-4fa4-ac73-3e6201da3239","Type":"ContainerDied","Data":"ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8"}
Sep 15 10:33:44.952203 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.952164    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2" event={"ID":"c9c7a156-ebf1-4fa4-ac73-3e6201da3239","Type":"ContainerDied","Data":"1a995718ad465a102666848032c3931a5c95dd77532e5081dcb15433a3e2a1fc"}
Sep 15 10:33:44.960878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.960858    2584 scope.go:117] "RemoveContainer" containerID="62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861"
Sep 15 10:33:44.961151 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:33:44.961132    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861\": container with ID starting with 62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861 not found: ID does not exist" containerID="62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861"
Sep 15 10:33:44.961206 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.961159    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861"} err="failed to get container status \"62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861\": rpc error: code = NotFound desc = could not find container \"62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861\": container with ID starting with 62a3680755864f9c410a7adad4242406f7b71051bd33e85dc265a7b27e662861 not found: ID does not exist"
Sep 15 10:33:44.961206 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.961176    2584 scope.go:117] "RemoveContainer" containerID="ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8"
Sep 15 10:33:44.969538 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.969522    2584 scope.go:117] "RemoveContainer" containerID="ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8"
Sep 15 10:33:44.969793 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:33:44.969772    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8\": container with ID starting with ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8 not found: ID does not exist" containerID="ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8"
Sep 15 10:33:44.969878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.969817    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8"} err="failed to get container status \"ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8\": rpc error: code = NotFound desc = could not find container \"ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8\": container with ID starting with ef610b8ecc3c293f614b50c3e292923ec8c813599b499c260bab2466c129a0a8 not found: ID does not exist"
Sep 15 10:33:44.976433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.976384    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"]
Sep 15 10:33:44.982078 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.982057    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8110bc-6596b4f65f-ckttj"]
Sep 15 10:33:44.991847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.991815    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"]
Sep 15 10:33:44.995557 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:44.995534    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8110bc-54bf5bcb98-njlx2"]
Sep 15 10:33:45.786500 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:45.786468    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9c7a156-ebf1-4fa4-ac73-3e6201da3239" path="/var/lib/kubelet/pods/c9c7a156-ebf1-4fa4-ac73-3e6201da3239/volumes"
Sep 15 10:33:45.786900 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:45.786883    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f336409e-2a24-4b0e-863f-c496baeacf99" path="/var/lib/kubelet/pods/f336409e-2a24-4b0e-863f-c496baeacf99/volumes"
Sep 15 10:33:49.945054 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:49.945017    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"]
Sep 15 10:33:50.956625 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:33:50.956578    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:34:02.161835 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.161740    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"]
Sep 15 10:34:02.162322 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.162299    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c7a156-ebf1-4fa4-ac73-3e6201da3239" containerName="payload-pre-processing"
Sep 15 10:34:02.162367 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.162327    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c7a156-ebf1-4fa4-ac73-3e6201da3239" containerName="payload-pre-processing"
Sep 15 10:34:02.162402 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.162367    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f336409e-2a24-4b0e-863f-c496baeacf99" containerName="payload-processing"
Sep 15 10:34:02.162402 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.162377    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f336409e-2a24-4b0e-863f-c496baeacf99" containerName="payload-processing"
Sep 15 10:34:02.162514 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.162502    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c7a156-ebf1-4fa4-ac73-3e6201da3239" containerName="payload-pre-processing"
Sep 15 10:34:02.162554 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.162520    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f336409e-2a24-4b0e-863f-c496baeacf99" containerName="payload-processing"
Sep 15 10:34:02.167255 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.167235    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.169312 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.169290    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8c1047\"/\"premium-w0-8c1047-kserve-self-signed-certs\""
Sep 15 10:34:02.174888 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.174794    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"]
Sep 15 10:34:02.244008 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.243964    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tls-certs\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.244008 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.244015    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-model-cache\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.244239 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.244041    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-home\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.244239 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.244095    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-dshm\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.244239 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.244131    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tmp-dir\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.244239 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.244214    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkn8v\" (UniqueName: \"kubernetes.io/projected/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-kube-api-access-rkn8v\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.345254 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.345211    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-model-cache\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.345254 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.345255    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-home\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.345466 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.345277    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-dshm\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.345466 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.345296    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tmp-dir\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.345466 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.345411    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkn8v\" (UniqueName: \"kubernetes.io/projected/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-kube-api-access-rkn8v\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.345598 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.345505    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tls-certs\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.345727 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.345704    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-model-cache\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.345834 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.345780    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-home\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.345957 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.345865    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tmp-dir\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.347664 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.347630    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-dshm\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.347971 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.347953    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tls-certs\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.353099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.353077    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkn8v\" (UniqueName: \"kubernetes.io/projected/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-kube-api-access-rkn8v\") pod \"premium-w0-8c1047-kserve-666dbd5487-jwtwl\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") " pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.477737 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.477635    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:02.747721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:02.747634    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"]
Sep 15 10:34:02.749945 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:34:02.749918    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29d873a2_0f82_45a3_9fcf_cf193bde0e8c.slice/crio-aaa460e5315ca8d209354e752ac5b748c4dbf76ba0181c460a4ba62f6f63ed0e WatchSource:0}: Error finding container aaa460e5315ca8d209354e752ac5b748c4dbf76ba0181c460a4ba62f6f63ed0e: Status 404 returned error can't find the container with id aaa460e5315ca8d209354e752ac5b748c4dbf76ba0181c460a4ba62f6f63ed0e
Sep 15 10:34:03.041075 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:03.040984    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl" event={"ID":"29d873a2-0f82-45a3-9fcf-cf193bde0e8c","Type":"ContainerStarted","Data":"2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002"}
Sep 15 10:34:03.041075 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:03.041032    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl" event={"ID":"29d873a2-0f82-45a3-9fcf-cf193bde0e8c","Type":"ContainerStarted","Data":"aaa460e5315ca8d209354e752ac5b748c4dbf76ba0181c460a4ba62f6f63ed0e"}
Sep 15 10:34:03.059601 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:03.059548    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl" podStartSLOduration=1.059532171 podStartE2EDuration="1.059532171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:34:03.058555666 +0000 UTC m=+961.752711348" watchObservedRunningTime="2026-09-15 10:34:03.059532171 +0000 UTC m=+961.753687863"
Sep 15 10:34:04.964946 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:04.964886    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9" podUID="982f2f7e-9c16-429c-9481-744cd4cb712a" containerName="main" containerID="cri-o://510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e" gracePeriod=45
Sep 15 10:34:05.219388 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.219330    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:34:05.275083 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275050    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-tmp-dir\") pod \"982f2f7e-9c16-429c-9481-744cd4cb712a\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") "
Sep 15 10:34:05.275257 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275105    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982f2f7e-9c16-429c-9481-744cd4cb712a-tls-certs\") pod \"982f2f7e-9c16-429c-9481-744cd4cb712a\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") "
Sep 15 10:34:05.275257 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275133    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-model-cache\") pod \"982f2f7e-9c16-429c-9481-744cd4cb712a\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") "
Sep 15 10:34:05.275257 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275162    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wwtv\" (UniqueName: \"kubernetes.io/projected/982f2f7e-9c16-429c-9481-744cd4cb712a-kube-api-access-9wwtv\") pod \"982f2f7e-9c16-429c-9481-744cd4cb712a\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") "
Sep 15 10:34:05.275257 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275233    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-dshm\") pod \"982f2f7e-9c16-429c-9481-744cd4cb712a\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") "
Sep 15 10:34:05.275454 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275285    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-home\") pod \"982f2f7e-9c16-429c-9481-744cd4cb712a\" (UID: \"982f2f7e-9c16-429c-9481-744cd4cb712a\") "
Sep 15 10:34:05.275454 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275406    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "982f2f7e-9c16-429c-9481-744cd4cb712a" (UID: "982f2f7e-9c16-429c-9481-744cd4cb712a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:34:05.275566 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275445    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-model-cache" (OuterVolumeSpecName: "model-cache") pod "982f2f7e-9c16-429c-9481-744cd4cb712a" (UID: "982f2f7e-9c16-429c-9481-744cd4cb712a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:34:05.275630 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275562    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-home" (OuterVolumeSpecName: "home") pod "982f2f7e-9c16-429c-9481-744cd4cb712a" (UID: "982f2f7e-9c16-429c-9481-744cd4cb712a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:34:05.275689 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275673    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:34:05.275730 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275699    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:34:05.275730 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.275716    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:34:05.277391 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.277367    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-dshm" (OuterVolumeSpecName: "dshm") pod "982f2f7e-9c16-429c-9481-744cd4cb712a" (UID: "982f2f7e-9c16-429c-9481-744cd4cb712a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:34:05.277495 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.277475    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982f2f7e-9c16-429c-9481-744cd4cb712a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "982f2f7e-9c16-429c-9481-744cd4cb712a" (UID: "982f2f7e-9c16-429c-9481-744cd4cb712a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:34:05.277541 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.277493    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982f2f7e-9c16-429c-9481-744cd4cb712a-kube-api-access-9wwtv" (OuterVolumeSpecName: "kube-api-access-9wwtv") pod "982f2f7e-9c16-429c-9481-744cd4cb712a" (UID: "982f2f7e-9c16-429c-9481-744cd4cb712a"). InnerVolumeSpecName "kube-api-access-9wwtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:34:05.376364 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.376317    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982f2f7e-9c16-429c-9481-744cd4cb712a-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:34:05.376364 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.376360    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wwtv\" (UniqueName: \"kubernetes.io/projected/982f2f7e-9c16-429c-9481-744cd4cb712a-kube-api-access-9wwtv\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:34:05.376364 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:05.376369    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982f2f7e-9c16-429c-9481-744cd4cb712a-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:34:06.058574 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:06.058484    2584 generic.go:356] "Generic (PLEG): container finished" podID="982f2f7e-9c16-429c-9481-744cd4cb712a" containerID="510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e" exitCode=0
Sep 15 10:34:06.058574 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:06.058531    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"
Sep 15 10:34:06.059031 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:06.058569    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9" event={"ID":"982f2f7e-9c16-429c-9481-744cd4cb712a","Type":"ContainerDied","Data":"510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e"}
Sep 15 10:34:06.059031 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:06.058613    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9" event={"ID":"982f2f7e-9c16-429c-9481-744cd4cb712a","Type":"ContainerDied","Data":"9c144faef36b90e00255dbf25dc64174db5d3f1c4cdb7611ae17bf5b44c2a8be"}
Sep 15 10:34:06.059031 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:06.058630    2584 scope.go:117] "RemoveContainer" containerID="510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e"
Sep 15 10:34:06.066986 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:06.066960    2584 scope.go:117] "RemoveContainer" containerID="510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e"
Sep 15 10:34:06.067254 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:34:06.067232    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e\": container with ID starting with 510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e not found: ID does not exist" containerID="510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e"
Sep 15 10:34:06.067306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:06.067261    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e"} err="failed to get container status \"510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e\": rpc error: code = NotFound desc = could not find container \"510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e\": container with ID starting with 510c53ed45bdf5b33f309e5c4ecfa5cc163cb289c775435f530412c785b55a4e not found: ID does not exist"
Sep 15 10:34:06.074948 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:06.074926    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"]
Sep 15 10:34:06.078385 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:06.078364    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-8110bc/autores-model-8110bc-kserve-7f6dfbf6c9-gnch9"]
Sep 15 10:34:07.785340 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:07.785307    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="982f2f7e-9c16-429c-9481-744cd4cb712a" path="/var/lib/kubelet/pods/982f2f7e-9c16-429c-9481-744cd4cb712a/volumes"
Sep 15 10:34:08.860874 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:08.860837    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:34:12.478442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:12.478410    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:12.478872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:12.478458    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:12.491193 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:12.491165    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:13.107874 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:13.107843    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:34:22.568461 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.568424    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"]
Sep 15 10:34:22.569047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.569026    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="982f2f7e-9c16-429c-9481-744cd4cb712a" containerName="main"
Sep 15 10:34:22.569097 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.569051    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="982f2f7e-9c16-429c-9481-744cd4cb712a" containerName="main"
Sep 15 10:34:22.569156 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.569144    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="982f2f7e-9c16-429c-9481-744cd4cb712a" containerName="main"
Sep 15 10:34:22.574203 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.574179    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.577052 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.577028    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ff0655\"/\"premium-w4-ff0655-kserve-self-signed-certs\""
Sep 15 10:34:22.578081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.578055    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"]
Sep 15 10:34:22.643061 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.643015    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-dshm\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.643061 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.643068    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3798e11-c021-4603-98d5-2c940a59df7e-tls-certs\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.643299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.643116    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-home\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.643299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.643171    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5m79\" (UniqueName: \"kubernetes.io/projected/e3798e11-c021-4603-98d5-2c940a59df7e-kube-api-access-j5m79\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.643299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.643196    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-tmp-dir\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.643299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.643261    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-model-cache\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.744047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.744004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5m79\" (UniqueName: \"kubernetes.io/projected/e3798e11-c021-4603-98d5-2c940a59df7e-kube-api-access-j5m79\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.744047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.744056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-tmp-dir\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.744308 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.744108    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-model-cache\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.744308 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.744152    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-dshm\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.744308 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.744248    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3798e11-c021-4603-98d5-2c940a59df7e-tls-certs\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.744308 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.744295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-home\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.744560 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.744532    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-tmp-dir\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.744702 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.744603    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-model-cache\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.744702 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.744617    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-home\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.746410 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.746383    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-dshm\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.746838 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.746815    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3798e11-c021-4603-98d5-2c940a59df7e-tls-certs\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.751463 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.751446    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5m79\" (UniqueName: \"kubernetes.io/projected/e3798e11-c021-4603-98d5-2c940a59df7e-kube-api-access-j5m79\") pod \"premium-w4-ff0655-kserve-6567f67c6d-j4nwt\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") " pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:22.885582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:22.885493    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:23.180314 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:23.180287    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"]
Sep 15 10:34:23.182429 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:34:23.182400    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3798e11_c021_4603_98d5_2c940a59df7e.slice/crio-c5c3fcfbf5df3934f37d32c1e7959602626c5cda6527fdb79e5249ecacb38680 WatchSource:0}: Error finding container c5c3fcfbf5df3934f37d32c1e7959602626c5cda6527fdb79e5249ecacb38680: Status 404 returned error can't find the container with id c5c3fcfbf5df3934f37d32c1e7959602626c5cda6527fdb79e5249ecacb38680
Sep 15 10:34:24.147381 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:24.147346    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt" event={"ID":"e3798e11-c021-4603-98d5-2c940a59df7e","Type":"ContainerStarted","Data":"f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a"}
Sep 15 10:34:24.147381 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:24.147389    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt" event={"ID":"e3798e11-c021-4603-98d5-2c940a59df7e","Type":"ContainerStarted","Data":"c5c3fcfbf5df3934f37d32c1e7959602626c5cda6527fdb79e5249ecacb38680"}
Sep 15 10:34:24.163713 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:24.163667    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt" podStartSLOduration=2.163645164 podStartE2EDuration="2.163645164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:34:24.162551753 +0000 UTC m=+982.856707433" watchObservedRunningTime="2026-09-15 10:34:24.163645164 +0000 UTC m=+982.857800845"
Sep 15 10:34:32.886598 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:32.886544    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:32.887067 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:32.886610    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:32.899404 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:32.899376    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:33.206249 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:33.206166    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:34:33.773125 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:33.773086    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:34:34.581828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.581778    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"]
Sep 15 10:34:34.585769 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.585746    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:34:34.588065 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.588040    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-aeb4e3-dockercfg-pwmj6\""
Sep 15 10:34:34.588206 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.588187    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-aeb4e3\""
Sep 15 10:34:34.600135 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.600110    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"]
Sep 15 10:34:34.655676 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.655637    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe4383e-949a-4e01-96b4-543385c6bb85-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz\" (UID: \"bfe4383e-949a-4e01-96b4-543385c6bb85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:34:34.655886 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.655713    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6lf\" (UniqueName: \"kubernetes.io/projected/bfe4383e-949a-4e01-96b4-543385c6bb85-kube-api-access-fh6lf\") pod \"payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz\" (UID: \"bfe4383e-949a-4e01-96b4-543385c6bb85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:34:34.756938 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.756898    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6lf\" (UniqueName: \"kubernetes.io/projected/bfe4383e-949a-4e01-96b4-543385c6bb85-kube-api-access-fh6lf\") pod \"payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz\" (UID: \"bfe4383e-949a-4e01-96b4-543385c6bb85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:34:34.757121 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.756997    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe4383e-949a-4e01-96b4-543385c6bb85-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz\" (UID: \"bfe4383e-949a-4e01-96b4-543385c6bb85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:34:34.757679 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.757659    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe4383e-949a-4e01-96b4-543385c6bb85-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz\" (UID: \"bfe4383e-949a-4e01-96b4-543385c6bb85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:34:34.764952 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.764926    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6lf\" (UniqueName: \"kubernetes.io/projected/bfe4383e-949a-4e01-96b4-543385c6bb85-kube-api-access-fh6lf\") pod \"payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz\" (UID: \"bfe4383e-949a-4e01-96b4-543385c6bb85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:34:34.896110 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:34.896020    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:34:35.164207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:35.164176    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"]
Sep 15 10:34:35.166769 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:34:35.166741    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe4383e_949a_4e01_96b4_543385c6bb85.slice/crio-1aeb02d3739dab6ed6115bdcf7151b0b5d70e88998117b0d9de9e7bae328a7a5 WatchSource:0}: Error finding container 1aeb02d3739dab6ed6115bdcf7151b0b5d70e88998117b0d9de9e7bae328a7a5: Status 404 returned error can't find the container with id 1aeb02d3739dab6ed6115bdcf7151b0b5d70e88998117b0d9de9e7bae328a7a5
Sep 15 10:34:35.206990 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:35.206955    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz" event={"ID":"bfe4383e-949a-4e01-96b4-543385c6bb85","Type":"ContainerStarted","Data":"1aeb02d3739dab6ed6115bdcf7151b0b5d70e88998117b0d9de9e7bae328a7a5"}
Sep 15 10:34:36.214242 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:36.214203    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz" event={"ID":"bfe4383e-949a-4e01-96b4-543385c6bb85","Type":"ContainerStarted","Data":"8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167"}
Sep 15 10:34:36.214852 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:36.214264    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:34:36.230026 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:36.229974    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz" podStartSLOduration=2.229957584 podStartE2EDuration="2.229957584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:34:36.228135774 +0000 UTC m=+994.922291455" watchObservedRunningTime="2026-09-15 10:34:36.229957584 +0000 UTC m=+994.924113264"
Sep 15 10:34:42.033350 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.033313    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"]
Sep 15 10:34:42.036775 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.036757    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:34:42.039698 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.039675    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8f3975b3\""
Sep 15 10:34:42.040060 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.040040    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8f3975b3-dockercfg-xppvc\""
Sep 15 10:34:42.047191 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.047168    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"]
Sep 15 10:34:42.069022 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.068988    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"]
Sep 15 10:34:42.072389 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.072373    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:34:42.081618 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.081591    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"]
Sep 15 10:34:42.124310 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.124262    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4be3eb8-f513-46f9-b686-b943718419ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv\" (UID: \"d4be3eb8-f513-46f9-b686-b943718419ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:34:42.124487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.124393    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgvjq\" (UniqueName: \"kubernetes.io/projected/3339efba-78da-4fa5-8a98-0642f96a1421-kube-api-access-wgvjq\") pod \"payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b\" (UID: \"3339efba-78da-4fa5-8a98-0642f96a1421\") " pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:34:42.124487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.124442    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlcl\" (UniqueName: \"kubernetes.io/projected/d4be3eb8-f513-46f9-b686-b943718419ff-kube-api-access-2tlcl\") pod \"payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv\" (UID: \"d4be3eb8-f513-46f9-b686-b943718419ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:34:42.124487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.124478    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3339efba-78da-4fa5-8a98-0642f96a1421-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b\" (UID: \"3339efba-78da-4fa5-8a98-0642f96a1421\") " pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:34:42.225669 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.225614    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4be3eb8-f513-46f9-b686-b943718419ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv\" (UID: \"d4be3eb8-f513-46f9-b686-b943718419ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:34:42.225906 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.225752    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgvjq\" (UniqueName: \"kubernetes.io/projected/3339efba-78da-4fa5-8a98-0642f96a1421-kube-api-access-wgvjq\") pod \"payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b\" (UID: \"3339efba-78da-4fa5-8a98-0642f96a1421\") " pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:34:42.225906 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.225824    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlcl\" (UniqueName: \"kubernetes.io/projected/d4be3eb8-f513-46f9-b686-b943718419ff-kube-api-access-2tlcl\") pod \"payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv\" (UID: \"d4be3eb8-f513-46f9-b686-b943718419ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:34:42.225906 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.225861    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3339efba-78da-4fa5-8a98-0642f96a1421-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b\" (UID: \"3339efba-78da-4fa5-8a98-0642f96a1421\") " pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:34:42.226635 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.226607    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4be3eb8-f513-46f9-b686-b943718419ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv\" (UID: \"d4be3eb8-f513-46f9-b686-b943718419ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:34:42.226705 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.226658    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3339efba-78da-4fa5-8a98-0642f96a1421-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b\" (UID: \"3339efba-78da-4fa5-8a98-0642f96a1421\") " pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:34:42.233765 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.233740    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlcl\" (UniqueName: \"kubernetes.io/projected/d4be3eb8-f513-46f9-b686-b943718419ff-kube-api-access-2tlcl\") pod \"payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv\" (UID: \"d4be3eb8-f513-46f9-b686-b943718419ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:34:42.233880 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.233751    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgvjq\" (UniqueName: \"kubernetes.io/projected/3339efba-78da-4fa5-8a98-0642f96a1421-kube-api-access-wgvjq\") pod \"payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b\" (UID: \"3339efba-78da-4fa5-8a98-0642f96a1421\") " pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:34:42.348449 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.348355    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:34:42.383350 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.383315    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:34:42.624365 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.624335    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"]
Sep 15 10:34:42.626385 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:34:42.626358    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4be3eb8_f513_46f9_b686_b943718419ff.slice/crio-5ddce25d74d29e8e5b9fa884b96a246156ebc30e8b8be50992b6f1725cfb51f2 WatchSource:0}: Error finding container 5ddce25d74d29e8e5b9fa884b96a246156ebc30e8b8be50992b6f1725cfb51f2: Status 404 returned error can't find the container with id 5ddce25d74d29e8e5b9fa884b96a246156ebc30e8b8be50992b6f1725cfb51f2
Sep 15 10:34:42.666460 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:42.666437    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"]
Sep 15 10:34:42.672889 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:34:42.672864    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3339efba_78da_4fa5_8a98_0642f96a1421.slice/crio-3a62748f391c5e8f5c9fe6b9d653eb2dbbe92b1beac57dbb7848f2fcf45026e8 WatchSource:0}: Error finding container 3a62748f391c5e8f5c9fe6b9d653eb2dbbe92b1beac57dbb7848f2fcf45026e8: Status 404 returned error can't find the container with id 3a62748f391c5e8f5c9fe6b9d653eb2dbbe92b1beac57dbb7848f2fcf45026e8
Sep 15 10:34:43.248120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:43.248086    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv" event={"ID":"d4be3eb8-f513-46f9-b686-b943718419ff","Type":"ContainerStarted","Data":"7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f"}
Sep 15 10:34:43.248560 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:43.248138    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv" event={"ID":"d4be3eb8-f513-46f9-b686-b943718419ff","Type":"ContainerStarted","Data":"5ddce25d74d29e8e5b9fa884b96a246156ebc30e8b8be50992b6f1725cfb51f2"}
Sep 15 10:34:43.248560 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:43.248198    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:34:43.249655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:43.249632    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b" event={"ID":"3339efba-78da-4fa5-8a98-0642f96a1421","Type":"ContainerStarted","Data":"7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c"}
Sep 15 10:34:43.249778 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:43.249674    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b" event={"ID":"3339efba-78da-4fa5-8a98-0642f96a1421","Type":"ContainerStarted","Data":"3a62748f391c5e8f5c9fe6b9d653eb2dbbe92b1beac57dbb7848f2fcf45026e8"}
Sep 15 10:34:43.249778 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:43.249757    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:34:43.262530 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:43.262478    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv" podStartSLOduration=1.262460699 podStartE2EDuration="1.262460699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:34:43.261583679 +0000 UTC m=+1001.955739362" watchObservedRunningTime="2026-09-15 10:34:43.262460699 +0000 UTC m=+1001.956616380"
Sep 15 10:34:43.277404 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:43.277359    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b" podStartSLOduration=1.277340502 podStartE2EDuration="1.277340502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:34:43.276770287 +0000 UTC m=+1001.970925967" watchObservedRunningTime="2026-09-15 10:34:43.277340502 +0000 UTC m=+1001.971496173"
Sep 15 10:34:47.221816 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:47.221766    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:34:54.256609 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:54.256576    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:34:54.257044 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:34:54.256642    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:35:08.962108 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:08.962066    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:35:09.873637 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:09.873596    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"]
Sep 15 10:35:09.878349 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:09.878325    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"
Sep 15 10:35:09.880864 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:09.880839    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jrzm\""
Sep 15 10:35:09.880999 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:09.880843    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 10:35:09.884489 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:09.884466    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"]
Sep 15 10:35:09.894919 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:09.894894    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/450112fa-20a0-46e3-a925-8324318184a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm\" (UID: \"450112fa-20a0-46e3-a925-8324318184a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"
Sep 15 10:35:09.995412 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:09.995374    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/450112fa-20a0-46e3-a925-8324318184a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm\" (UID: \"450112fa-20a0-46e3-a925-8324318184a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"
Sep 15 10:35:09.996047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:09.996026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/450112fa-20a0-46e3-a925-8324318184a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm\" (UID: \"450112fa-20a0-46e3-a925-8324318184a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"
Sep 15 10:35:10.190838 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:10.190718    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"
Sep 15 10:35:10.464635 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:35:10.464605    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450112fa_20a0_46e3_a925_8324318184a3.slice/crio-9dc609591fc80fe6867ab99cabfa02acb9e1578cf6bf36be1eec05687b314de6 WatchSource:0}: Error finding container 9dc609591fc80fe6867ab99cabfa02acb9e1578cf6bf36be1eec05687b314de6: Status 404 returned error can't find the container with id 9dc609591fc80fe6867ab99cabfa02acb9e1578cf6bf36be1eec05687b314de6
Sep 15 10:35:10.468603 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:10.468579    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"]
Sep 15 10:35:11.390379 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:11.390348    2584 generic.go:356] "Generic (PLEG): container finished" podID="450112fa-20a0-46e3-a925-8324318184a3" containerID="0c738df31376516a867ab9ce985729829395d05b7e323d0efd3e634936862354" exitCode=0
Sep 15 10:35:11.390755 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:11.390438    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm" event={"ID":"450112fa-20a0-46e3-a925-8324318184a3","Type":"ContainerDied","Data":"0c738df31376516a867ab9ce985729829395d05b7e323d0efd3e634936862354"}
Sep 15 10:35:11.390755 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:11.390488    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm" event={"ID":"450112fa-20a0-46e3-a925-8324318184a3","Type":"ContainerStarted","Data":"9dc609591fc80fe6867ab99cabfa02acb9e1578cf6bf36be1eec05687b314de6"}
Sep 15 10:35:12.533460 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:12.533436    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"
Sep 15 10:35:12.628129 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:12.628099    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/450112fa-20a0-46e3-a925-8324318184a3-maas-api-service-ca\") pod \"450112fa-20a0-46e3-a925-8324318184a3\" (UID: \"450112fa-20a0-46e3-a925-8324318184a3\") "
Sep 15 10:35:12.628450 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:12.628425    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450112fa-20a0-46e3-a925-8324318184a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "450112fa-20a0-46e3-a925-8324318184a3" (UID: "450112fa-20a0-46e3-a925-8324318184a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:35:12.729126 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:12.729033    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/450112fa-20a0-46e3-a925-8324318184a3-maas-api-service-ca\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:35:13.400973 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:13.400943    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"
Sep 15 10:35:13.401179 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:13.400942    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm" event={"ID":"450112fa-20a0-46e3-a925-8324318184a3","Type":"ContainerDied","Data":"9dc609591fc80fe6867ab99cabfa02acb9e1578cf6bf36be1eec05687b314de6"}
Sep 15 10:35:13.401179 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:13.401067    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dc609591fc80fe6867ab99cabfa02acb9e1578cf6bf36be1eec05687b314de6"
Sep 15 10:35:20.414610 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.414572    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"]
Sep 15 10:35:20.415081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.414874    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b" podUID="3339efba-78da-4fa5-8a98-0642f96a1421" containerName="payload-processing" containerID="cri-o://7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c" gracePeriod=30
Sep 15 10:35:20.420911 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.420884    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"]
Sep 15 10:35:20.421138 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.421097    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv" podUID="d4be3eb8-f513-46f9-b686-b943718419ff" containerName="payload-pre-processing" containerID="cri-o://7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f" gracePeriod=30
Sep 15 10:35:20.562131 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.562076    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"]
Sep 15 10:35:20.570549 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.570516    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f3975b3-9dlsm"]
Sep 15 10:35:20.695651 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.695607    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:35:20.698530 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.698509    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:35:20.804946 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.804911    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3339efba-78da-4fa5-8a98-0642f96a1421-plugins-config-volume\") pod \"3339efba-78da-4fa5-8a98-0642f96a1421\" (UID: \"3339efba-78da-4fa5-8a98-0642f96a1421\") "
Sep 15 10:35:20.805201 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.804973    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4be3eb8-f513-46f9-b686-b943718419ff-plugins-config-volume\") pod \"d4be3eb8-f513-46f9-b686-b943718419ff\" (UID: \"d4be3eb8-f513-46f9-b686-b943718419ff\") "
Sep 15 10:35:20.805201 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.805015    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tlcl\" (UniqueName: \"kubernetes.io/projected/d4be3eb8-f513-46f9-b686-b943718419ff-kube-api-access-2tlcl\") pod \"d4be3eb8-f513-46f9-b686-b943718419ff\" (UID: \"d4be3eb8-f513-46f9-b686-b943718419ff\") "
Sep 15 10:35:20.805201 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.805074    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgvjq\" (UniqueName: \"kubernetes.io/projected/3339efba-78da-4fa5-8a98-0642f96a1421-kube-api-access-wgvjq\") pod \"3339efba-78da-4fa5-8a98-0642f96a1421\" (UID: \"3339efba-78da-4fa5-8a98-0642f96a1421\") "
Sep 15 10:35:20.805400 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.805354    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3339efba-78da-4fa5-8a98-0642f96a1421-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3339efba-78da-4fa5-8a98-0642f96a1421" (UID: "3339efba-78da-4fa5-8a98-0642f96a1421"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:35:20.805400 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.805357    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4be3eb8-f513-46f9-b686-b943718419ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4be3eb8-f513-46f9-b686-b943718419ff" (UID: "d4be3eb8-f513-46f9-b686-b943718419ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:35:20.805510 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.805463    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3339efba-78da-4fa5-8a98-0642f96a1421-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:35:20.805510 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.805484    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4be3eb8-f513-46f9-b686-b943718419ff-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:35:20.807730 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.807702    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4be3eb8-f513-46f9-b686-b943718419ff-kube-api-access-2tlcl" (OuterVolumeSpecName: "kube-api-access-2tlcl") pod "d4be3eb8-f513-46f9-b686-b943718419ff" (UID: "d4be3eb8-f513-46f9-b686-b943718419ff"). InnerVolumeSpecName "kube-api-access-2tlcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:35:20.807847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.807735    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3339efba-78da-4fa5-8a98-0642f96a1421-kube-api-access-wgvjq" (OuterVolumeSpecName: "kube-api-access-wgvjq") pod "3339efba-78da-4fa5-8a98-0642f96a1421" (UID: "3339efba-78da-4fa5-8a98-0642f96a1421"). InnerVolumeSpecName "kube-api-access-wgvjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:35:20.906342 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.906311    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tlcl\" (UniqueName: \"kubernetes.io/projected/d4be3eb8-f513-46f9-b686-b943718419ff-kube-api-access-2tlcl\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:35:20.906342 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:20.906338    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgvjq\" (UniqueName: \"kubernetes.io/projected/3339efba-78da-4fa5-8a98-0642f96a1421-kube-api-access-wgvjq\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:35:21.446607 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.446574    2584 generic.go:356] "Generic (PLEG): container finished" podID="d4be3eb8-f513-46f9-b686-b943718419ff" containerID="7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f" exitCode=0
Sep 15 10:35:21.447060 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.446619    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"
Sep 15 10:35:21.447060 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.446658    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv" event={"ID":"d4be3eb8-f513-46f9-b686-b943718419ff","Type":"ContainerDied","Data":"7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f"}
Sep 15 10:35:21.447060 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.446720    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv" event={"ID":"d4be3eb8-f513-46f9-b686-b943718419ff","Type":"ContainerDied","Data":"5ddce25d74d29e8e5b9fa884b96a246156ebc30e8b8be50992b6f1725cfb51f2"}
Sep 15 10:35:21.447060 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.446755    2584 scope.go:117] "RemoveContainer" containerID="7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f"
Sep 15 10:35:21.448100 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.447989    2584 generic.go:356] "Generic (PLEG): container finished" podID="3339efba-78da-4fa5-8a98-0642f96a1421" containerID="7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c" exitCode=0
Sep 15 10:35:21.448100 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.448038    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"
Sep 15 10:35:21.448220 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.448080    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b" event={"ID":"3339efba-78da-4fa5-8a98-0642f96a1421","Type":"ContainerDied","Data":"7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c"}
Sep 15 10:35:21.448220 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.448147    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b" event={"ID":"3339efba-78da-4fa5-8a98-0642f96a1421","Type":"ContainerDied","Data":"3a62748f391c5e8f5c9fe6b9d653eb2dbbe92b1beac57dbb7848f2fcf45026e8"}
Sep 15 10:35:21.455872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.455854    2584 scope.go:117] "RemoveContainer" containerID="7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f"
Sep 15 10:35:21.456283 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:35:21.456143    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f\": container with ID starting with 7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f not found: ID does not exist" containerID="7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f"
Sep 15 10:35:21.456283 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.456171    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f"} err="failed to get container status \"7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f\": rpc error: code = NotFound desc = could not find container \"7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f\": container with ID starting with 7a6f2468f3b2cc1df57184ddc576bad4afebd1e538af7ccc5c3840e002aaf28f not found: ID does not exist"
Sep 15 10:35:21.456283 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.456191    2584 scope.go:117] "RemoveContainer" containerID="7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c"
Sep 15 10:35:21.465117 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.465073    2584 scope.go:117] "RemoveContainer" containerID="7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c"
Sep 15 10:35:21.465449 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:35:21.465424    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c\": container with ID starting with 7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c not found: ID does not exist" containerID="7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c"
Sep 15 10:35:21.465544 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.465457    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c"} err="failed to get container status \"7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c\": rpc error: code = NotFound desc = could not find container \"7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c\": container with ID starting with 7821cf3ab7945c3cd0b48d08c804f0a28c88903d6603c01bcd17bd7a8501c34c not found: ID does not exist"
Sep 15 10:35:21.472203 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.472181    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"]
Sep 15 10:35:21.476106 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.476083    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f3975b3-7759c96f8-bjgcv"]
Sep 15 10:35:21.487992 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.487969    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"]
Sep 15 10:35:21.496008 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.495985    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f3975b3-6d5c8775d9-lvw2b"]
Sep 15 10:35:21.788886 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.788190    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3339efba-78da-4fa5-8a98-0642f96a1421" path="/var/lib/kubelet/pods/3339efba-78da-4fa5-8a98-0642f96a1421/volumes"
Sep 15 10:35:21.788886 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.788718    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="450112fa-20a0-46e3-a925-8324318184a3" path="/var/lib/kubelet/pods/450112fa-20a0-46e3-a925-8324318184a3/volumes"
Sep 15 10:35:21.789523 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:21.789502    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4be3eb8-f513-46f9-b686-b943718419ff" path="/var/lib/kubelet/pods/d4be3eb8-f513-46f9-b686-b943718419ff/volumes"
Sep 15 10:35:28.456029 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:28.455994    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:35:47.869495 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.869459    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"]
Sep 15 10:35:47.869995 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.869851    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3339efba-78da-4fa5-8a98-0642f96a1421" containerName="payload-processing"
Sep 15 10:35:47.869995 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.869864    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3339efba-78da-4fa5-8a98-0642f96a1421" containerName="payload-processing"
Sep 15 10:35:47.869995 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.869883    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4be3eb8-f513-46f9-b686-b943718419ff" containerName="payload-pre-processing"
Sep 15 10:35:47.869995 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.869888    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4be3eb8-f513-46f9-b686-b943718419ff" containerName="payload-pre-processing"
Sep 15 10:35:47.869995 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.869896    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="450112fa-20a0-46e3-a925-8324318184a3" containerName="revoke-keys"
Sep 15 10:35:47.869995 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.869902    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="450112fa-20a0-46e3-a925-8324318184a3" containerName="revoke-keys"
Sep 15 10:35:47.869995 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.869976    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4be3eb8-f513-46f9-b686-b943718419ff" containerName="payload-pre-processing"
Sep 15 10:35:47.869995 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.869983    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="450112fa-20a0-46e3-a925-8324318184a3" containerName="revoke-keys"
Sep 15 10:35:47.869995 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.869993    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3339efba-78da-4fa5-8a98-0642f96a1421" containerName="payload-processing"
Sep 15 10:35:47.873264 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.873244    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:47.876242 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.876223    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dkr42\""
Sep 15 10:35:47.876778 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.876752    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 10:35:47.876924 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.876775    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 10:35:47.876924 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.876792    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-30651f-serving-cert\""
Sep 15 10:35:47.876924 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.876774    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 10:35:47.884212 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.884191    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"]
Sep 15 10:35:47.968501 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.968469    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:47.968655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.968530    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:47.968655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.968585    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:47.968655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:47.968611    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5ppv\" (UniqueName: \"kubernetes.io/projected/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-kube-api-access-z5ppv\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:48.069875 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:48.069837    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:48.070069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:48.069896    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5ppv\" (UniqueName: \"kubernetes.io/projected/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-kube-api-access-z5ppv\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:48.070069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:48.069951    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:48.070069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:48.070033    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:48.070252 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:35:48.070165    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-serving-cert: secret "maas-api-e2e-shared-a-w2-30651f-serving-cert" not found
Sep 15 10:35:48.070252 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:35:48.070248    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-maas-api-tls podName:2b5fe564-c4f1-4ad3-8d5d-75d90417cb28 nodeName:}" failed. No retries permitted until 2026-09-15 10:35:48.570227264 +0000 UTC m=+1067.264382926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-maas-api-tls") pod "maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf" (UID: "2b5fe564-c4f1-4ad3-8d5d-75d90417cb28") : secret "maas-api-e2e-shared-a-w2-30651f-serving-cert" not found
Sep 15 10:35:48.070472 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:48.070449    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:48.072375 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:48.072355    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:48.078865 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:48.078842    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5ppv\" (UniqueName: \"kubernetes.io/projected/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-kube-api-access-z5ppv\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:48.575046 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:48.575004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:48.577423 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:48.577402    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:48.791516 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:48.791479    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:49.061741 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:49.061710    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"]
Sep 15 10:35:49.065083 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:35:49.065049    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5fe564_c4f1_4ad3_8d5d_75d90417cb28.slice/crio-d096bb55f2da80e6a24a79e8dffc5641bcf5c9da9121bc349eee363edfec52f8 WatchSource:0}: Error finding container d096bb55f2da80e6a24a79e8dffc5641bcf5c9da9121bc349eee363edfec52f8: Status 404 returned error can't find the container with id d096bb55f2da80e6a24a79e8dffc5641bcf5c9da9121bc349eee363edfec52f8
Sep 15 10:35:49.066425 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:49.066408    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 10:35:49.594260 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:49.594221    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf" event={"ID":"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28","Type":"ContainerStarted","Data":"3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891"}
Sep 15 10:35:49.594433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:49.594272    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf" event={"ID":"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28","Type":"ContainerStarted","Data":"d096bb55f2da80e6a24a79e8dffc5641bcf5c9da9121bc349eee363edfec52f8"}
Sep 15 10:35:49.594433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:49.594357    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:49.611864 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:49.611817    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf" podStartSLOduration=2.219813076 podStartE2EDuration="2.611784838s" totalImagesPullingTime="391.971762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:35:49.609726923 +0000 UTC m=+1068.303882605" watchObservedRunningTime="2026-09-15 10:35:49.611784838 +0000 UTC m=+1068.305940547"
Sep 15 10:35:53.739689 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.739642    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"]
Sep 15 10:35:53.743300 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.743275    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.745672 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.745646    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-aeb4e3\"/\"openshift-service-ca.crt\""
Sep 15 10:35:53.745831 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.745719    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-aeb4e3\"/\"default-dockercfg-gfsdw\""
Sep 15 10:35:53.745916 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.745847    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-aeb4e3\"/\"kube-root-ca.crt\""
Sep 15 10:35:53.745983 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.745912    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-aeb4e3\"/\"explicit-model-aeb4e3-kserve-self-signed-certs\""
Sep 15 10:35:53.751230 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.750945    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"]
Sep 15 10:35:53.823444 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.823407    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m9fh\" (UniqueName: \"kubernetes.io/projected/f41e216d-f2db-4324-bcf9-e2b13b17edb7-kube-api-access-5m9fh\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.823612 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.823473    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tls-certs\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.823612 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.823532    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-model-cache\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.823775 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.823605    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-dshm\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.823775 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.823653    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tmp-dir\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.823775 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.823687    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-home\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.924463 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.924423    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tls-certs\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.924655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.924497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-model-cache\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.924655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.924519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-dshm\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.924655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.924535    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tmp-dir\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.924655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.924553    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-home\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.924655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.924596    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m9fh\" (UniqueName: \"kubernetes.io/projected/f41e216d-f2db-4324-bcf9-e2b13b17edb7-kube-api-access-5m9fh\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.925059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.925035    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-model-cache\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.925107 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.925035    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tmp-dir\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.925107 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.925075    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-home\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.927043 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.927022    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-dshm\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.927174 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.927102    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tls-certs\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:53.932248 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:53.932228    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m9fh\" (UniqueName: \"kubernetes.io/projected/f41e216d-f2db-4324-bcf9-e2b13b17edb7-kube-api-access-5m9fh\") pod \"explicit-model-aeb4e3-kserve-7cf49db797-qsrwb\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") " pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:54.054115 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:54.054080    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:35:54.324578 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:54.324552    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"]
Sep 15 10:35:54.327052 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:35:54.327019    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf41e216d_f2db_4324_bcf9_e2b13b17edb7.slice/crio-cee5f767fae9d8359ada60044ea7135ef8ae3678717eae10d079678af1c76860 WatchSource:0}: Error finding container cee5f767fae9d8359ada60044ea7135ef8ae3678717eae10d079678af1c76860: Status 404 returned error can't find the container with id cee5f767fae9d8359ada60044ea7135ef8ae3678717eae10d079678af1c76860
Sep 15 10:35:54.622682 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:54.622588    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb" event={"ID":"f41e216d-f2db-4324-bcf9-e2b13b17edb7","Type":"ContainerStarted","Data":"ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6"}
Sep 15 10:35:54.622682 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:54.622638    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb" event={"ID":"f41e216d-f2db-4324-bcf9-e2b13b17edb7","Type":"ContainerStarted","Data":"cee5f767fae9d8359ada60044ea7135ef8ae3678717eae10d079678af1c76860"}
Sep 15 10:35:54.641545 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:54.641487    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb" podStartSLOduration=1.6414727359999999 podStartE2EDuration="1.641472736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:35:54.640873822 +0000 UTC m=+1073.335029506" watchObservedRunningTime="2026-09-15 10:35:54.641472736 +0000 UTC m=+1073.335628410"
Sep 15 10:35:55.604585 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:55.604554    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:35:57.563393 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:35:57.563358    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:36:04.054952 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:04.054904    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:36:04.055431 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:04.054974    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:36:04.067731 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:04.067702    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:36:04.683163 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:04.683133    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:36:21.561215 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:21.561179    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:36:22.015488 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.015458    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"]
Sep 15 10:36:22.020094 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.020078    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:36:22.022660 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.022604    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ecf56f\""
Sep 15 10:36:22.022795 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.022704    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ecf56f-dockercfg-lvrcv\""
Sep 15 10:36:22.029065 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.029038    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"]
Sep 15 10:36:22.106782 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.106748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67p7\" (UniqueName: \"kubernetes.io/projected/5a6f2956-fd66-4701-be8b-696e178d230e-kube-api-access-w67p7\") pod \"payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj\" (UID: \"5a6f2956-fd66-4701-be8b-696e178d230e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:36:22.106967 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.106828    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6f2956-fd66-4701-be8b-696e178d230e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj\" (UID: \"5a6f2956-fd66-4701-be8b-696e178d230e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:36:22.207592 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.207558    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67p7\" (UniqueName: \"kubernetes.io/projected/5a6f2956-fd66-4701-be8b-696e178d230e-kube-api-access-w67p7\") pod \"payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj\" (UID: \"5a6f2956-fd66-4701-be8b-696e178d230e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:36:22.207769 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.207601    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6f2956-fd66-4701-be8b-696e178d230e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj\" (UID: \"5a6f2956-fd66-4701-be8b-696e178d230e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:36:22.208339 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.208319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6f2956-fd66-4701-be8b-696e178d230e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj\" (UID: \"5a6f2956-fd66-4701-be8b-696e178d230e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:36:22.215118 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.215096    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67p7\" (UniqueName: \"kubernetes.io/projected/5a6f2956-fd66-4701-be8b-696e178d230e-kube-api-access-w67p7\") pod \"payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj\" (UID: \"5a6f2956-fd66-4701-be8b-696e178d230e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:36:22.331551 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.331462    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:36:22.605964 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.605938    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"]
Sep 15 10:36:22.609091 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:36:22.609061    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6f2956_fd66_4701_be8b_696e178d230e.slice/crio-9fd0039851597111586b1524a13e5d2c9713c14df2e3d97b5e192ad98da7f8b9 WatchSource:0}: Error finding container 9fd0039851597111586b1524a13e5d2c9713c14df2e3d97b5e192ad98da7f8b9: Status 404 returned error can't find the container with id 9fd0039851597111586b1524a13e5d2c9713c14df2e3d97b5e192ad98da7f8b9
Sep 15 10:36:22.761442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.761404    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj" event={"ID":"5a6f2956-fd66-4701-be8b-696e178d230e","Type":"ContainerStarted","Data":"0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247"}
Sep 15 10:36:22.761622 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.761465    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj" event={"ID":"5a6f2956-fd66-4701-be8b-696e178d230e","Type":"ContainerStarted","Data":"9fd0039851597111586b1524a13e5d2c9713c14df2e3d97b5e192ad98da7f8b9"}
Sep 15 10:36:22.761622 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.761509    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:36:22.777197 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:22.777150    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj" podStartSLOduration=1.7771366450000001 podStartE2EDuration="1.777136645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:36:22.775234463 +0000 UTC m=+1101.469390148" watchObservedRunningTime="2026-09-15 10:36:22.777136645 +0000 UTC m=+1101.471292352"
Sep 15 10:36:33.769847 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:33.769791    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:36:37.658051 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:37.658016    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7879-aa46f1"]
Sep 15 10:36:37.661430 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:37.661413    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-aa46f1"
Sep 15 10:36:37.664026 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:37.664009    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f7bvs\""
Sep 15 10:36:37.667170 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:37.667148    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-aa46f1"]
Sep 15 10:36:37.766356 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:37.766317    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfxvx\" (UniqueName: \"kubernetes.io/projected/9db057a6-2448-4e7d-81b8-c56ae69c1639-kube-api-access-xfxvx\") pod \"test-curl-7879-aa46f1\" (UID: \"9db057a6-2448-4e7d-81b8-c56ae69c1639\") " pod="opendatahub/test-curl-7879-aa46f1"
Sep 15 10:36:37.867895 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:37.867859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfxvx\" (UniqueName: \"kubernetes.io/projected/9db057a6-2448-4e7d-81b8-c56ae69c1639-kube-api-access-xfxvx\") pod \"test-curl-7879-aa46f1\" (UID: \"9db057a6-2448-4e7d-81b8-c56ae69c1639\") " pod="opendatahub/test-curl-7879-aa46f1"
Sep 15 10:36:37.877207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:37.877174    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfxvx\" (UniqueName: \"kubernetes.io/projected/9db057a6-2448-4e7d-81b8-c56ae69c1639-kube-api-access-xfxvx\") pod \"test-curl-7879-aa46f1\" (UID: \"9db057a6-2448-4e7d-81b8-c56ae69c1639\") " pod="opendatahub/test-curl-7879-aa46f1"
Sep 15 10:36:37.972779 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:37.972696    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-aa46f1"
Sep 15 10:36:38.242575 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:38.242549    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-aa46f1"]
Sep 15 10:36:38.244895 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:36:38.244863    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9db057a6_2448_4e7d_81b8_c56ae69c1639.slice/crio-4221ea7b984050f3e41889e3fbd79ec09bcd796923f675eb9b24b358fb3edb9a WatchSource:0}: Error finding container 4221ea7b984050f3e41889e3fbd79ec09bcd796923f675eb9b24b358fb3edb9a: Status 404 returned error can't find the container with id 4221ea7b984050f3e41889e3fbd79ec09bcd796923f675eb9b24b358fb3edb9a
Sep 15 10:36:38.841632 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:38.841594    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-aa46f1" event={"ID":"9db057a6-2448-4e7d-81b8-c56ae69c1639","Type":"ContainerStarted","Data":"df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4"}
Sep 15 10:36:38.842031 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:38.841645    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-aa46f1" event={"ID":"9db057a6-2448-4e7d-81b8-c56ae69c1639","Type":"ContainerStarted","Data":"4221ea7b984050f3e41889e3fbd79ec09bcd796923f675eb9b24b358fb3edb9a"}
Sep 15 10:36:38.856627 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:38.856565    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7879-aa46f1" podStartSLOduration=1.85654801 podStartE2EDuration="1.85654801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:36:38.855064063 +0000 UTC m=+1117.549219744" watchObservedRunningTime="2026-09-15 10:36:38.85654801 +0000 UTC m=+1117.550703690"
Sep 15 10:36:38.873653 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:38.873619    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"]
Sep 15 10:36:38.874259 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:38.873879    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz" podUID="bfe4383e-949a-4e01-96b4-543385c6bb85" containerName="payload-pre-processing" containerID="cri-o://8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167" gracePeriod=30
Sep 15 10:36:39.133965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.133938    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:36:39.283783 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.283754    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe4383e-949a-4e01-96b4-543385c6bb85-plugins-config-volume\") pod \"bfe4383e-949a-4e01-96b4-543385c6bb85\" (UID: \"bfe4383e-949a-4e01-96b4-543385c6bb85\") "
Sep 15 10:36:39.283976 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.283808    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh6lf\" (UniqueName: \"kubernetes.io/projected/bfe4383e-949a-4e01-96b4-543385c6bb85-kube-api-access-fh6lf\") pod \"bfe4383e-949a-4e01-96b4-543385c6bb85\" (UID: \"bfe4383e-949a-4e01-96b4-543385c6bb85\") "
Sep 15 10:36:39.284142 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.284118    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe4383e-949a-4e01-96b4-543385c6bb85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfe4383e-949a-4e01-96b4-543385c6bb85" (UID: "bfe4383e-949a-4e01-96b4-543385c6bb85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:36:39.285914 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.285885    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe4383e-949a-4e01-96b4-543385c6bb85-kube-api-access-fh6lf" (OuterVolumeSpecName: "kube-api-access-fh6lf") pod "bfe4383e-949a-4e01-96b4-543385c6bb85" (UID: "bfe4383e-949a-4e01-96b4-543385c6bb85"). InnerVolumeSpecName "kube-api-access-fh6lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:36:39.385173 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.385090    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe4383e-949a-4e01-96b4-543385c6bb85-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:36:39.385173 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.385123    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6lf\" (UniqueName: \"kubernetes.io/projected/bfe4383e-949a-4e01-96b4-543385c6bb85-kube-api-access-fh6lf\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:36:39.446142 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.446105    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7879-aa46f1"]
Sep 15 10:36:39.452178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.452142    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7879-aa46f1"]
Sep 15 10:36:39.848265 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.848232    2584 generic.go:356] "Generic (PLEG): container finished" podID="bfe4383e-949a-4e01-96b4-543385c6bb85" containerID="8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167" exitCode=0
Sep 15 10:36:39.848723 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.848279    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"
Sep 15 10:36:39.848723 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.848317    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz" event={"ID":"bfe4383e-949a-4e01-96b4-543385c6bb85","Type":"ContainerDied","Data":"8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167"}
Sep 15 10:36:39.848723 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.848370    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz" event={"ID":"bfe4383e-949a-4e01-96b4-543385c6bb85","Type":"ContainerDied","Data":"1aeb02d3739dab6ed6115bdcf7151b0b5d70e88998117b0d9de9e7bae328a7a5"}
Sep 15 10:36:39.848723 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.848393    2584 scope.go:117] "RemoveContainer" containerID="8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167"
Sep 15 10:36:39.848723 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:36:39.848532    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7879-aa46f1" container="test-curl-7879-aa46f1"
Sep 15 10:36:39.857431 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.857412    2584 scope.go:117] "RemoveContainer" containerID="8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167"
Sep 15 10:36:39.857685 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:36:39.857663    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167\": container with ID starting with 8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167 not found: ID does not exist" containerID="8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167"
Sep 15 10:36:39.857753 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.857693    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167"} err="failed to get container status \"8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167\": rpc error: code = NotFound desc = could not find container \"8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167\": container with ID starting with 8b041deded4c156337fb7e12ab04992770869bc2c2515dfd52b2352715f10167 not found: ID does not exist"
Sep 15 10:36:39.864361 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.864339    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"]
Sep 15 10:36:39.868704 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:39.868686    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aeb4e3-f4dd89486-c2cqz"]
Sep 15 10:36:40.854447 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:40.854412    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7879-aa46f1" podUID="9db057a6-2448-4e7d-81b8-c56ae69c1639" containerName="test-curl-7879-aa46f1" containerID="cri-o://df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4" gracePeriod=2
Sep 15 10:36:41.784932 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:41.784902    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe4383e-949a-4e01-96b4-543385c6bb85" path="/var/lib/kubelet/pods/bfe4383e-949a-4e01-96b4-543385c6bb85/volumes"
Sep 15 10:36:43.032754 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:43.032732    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-aa46f1"
Sep 15 10:36:43.120214 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:43.120128    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfxvx\" (UniqueName: \"kubernetes.io/projected/9db057a6-2448-4e7d-81b8-c56ae69c1639-kube-api-access-xfxvx\") pod \"9db057a6-2448-4e7d-81b8-c56ae69c1639\" (UID: \"9db057a6-2448-4e7d-81b8-c56ae69c1639\") "
Sep 15 10:36:43.122299 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:43.122271    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db057a6-2448-4e7d-81b8-c56ae69c1639-kube-api-access-xfxvx" (OuterVolumeSpecName: "kube-api-access-xfxvx") pod "9db057a6-2448-4e7d-81b8-c56ae69c1639" (UID: "9db057a6-2448-4e7d-81b8-c56ae69c1639"). InnerVolumeSpecName "kube-api-access-xfxvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:36:43.221855 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:43.221823    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfxvx\" (UniqueName: \"kubernetes.io/projected/9db057a6-2448-4e7d-81b8-c56ae69c1639-kube-api-access-xfxvx\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:36:43.785115 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:43.785083    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9db057a6-2448-4e7d-81b8-c56ae69c1639" path="/var/lib/kubelet/pods/9db057a6-2448-4e7d-81b8-c56ae69c1639/volumes"
Sep 15 10:36:43.870741 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:43.870701    2584 generic.go:356] "Generic (PLEG): container finished" podID="9db057a6-2448-4e7d-81b8-c56ae69c1639" containerID="df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4" exitCode=137
Sep 15 10:36:43.870741 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:43.870742    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-aa46f1"
Sep 15 10:36:43.870958 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:43.870810    2584 scope.go:117] "RemoveContainer" containerID="df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4"
Sep 15 10:36:43.879116 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:43.879098    2584 scope.go:117] "RemoveContainer" containerID="df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4"
Sep 15 10:36:43.879382 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:36:43.879363    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4\": container with ID starting with df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4 not found: ID does not exist" containerID="df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4"
Sep 15 10:36:43.879449 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:43.879394    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4"} err="failed to get container status \"df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4\": rpc error: code = NotFound desc = could not find container \"df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4\": container with ID starting with df0ac46efd66a2a3fc1f64209536162a3aba6201f99590b60a11de8d290882b4 not found: ID does not exist"
Sep 15 10:36:44.345093 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:44.345057    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"]
Sep 15 10:36:46.204830 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.204782    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7879-602306"]
Sep 15 10:36:46.205208 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.205155    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe4383e-949a-4e01-96b4-543385c6bb85" containerName="payload-pre-processing"
Sep 15 10:36:46.205208 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.205167    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe4383e-949a-4e01-96b4-543385c6bb85" containerName="payload-pre-processing"
Sep 15 10:36:46.205208 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.205184    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9db057a6-2448-4e7d-81b8-c56ae69c1639" containerName="test-curl-7879-aa46f1"
Sep 15 10:36:46.205208 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.205190    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9db057a6-2448-4e7d-81b8-c56ae69c1639" containerName="test-curl-7879-aa46f1"
Sep 15 10:36:46.205345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.205266    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9db057a6-2448-4e7d-81b8-c56ae69c1639" containerName="test-curl-7879-aa46f1"
Sep 15 10:36:46.205345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.205275    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe4383e-949a-4e01-96b4-543385c6bb85" containerName="payload-pre-processing"
Sep 15 10:36:46.209378 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.209361    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-602306"
Sep 15 10:36:46.212743 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.212715    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f7bvs\""
Sep 15 10:36:46.220149 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.220123    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-602306"]
Sep 15 10:36:46.353391 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.353353    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6jck\" (UniqueName: \"kubernetes.io/projected/7fad425d-7299-4e20-a756-0328f364a333-kube-api-access-x6jck\") pod \"test-curl-7879-602306\" (UID: \"7fad425d-7299-4e20-a756-0328f364a333\") " pod="opendatahub/test-curl-7879-602306"
Sep 15 10:36:46.454266 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.454228    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6jck\" (UniqueName: \"kubernetes.io/projected/7fad425d-7299-4e20-a756-0328f364a333-kube-api-access-x6jck\") pod \"test-curl-7879-602306\" (UID: \"7fad425d-7299-4e20-a756-0328f364a333\") " pod="opendatahub/test-curl-7879-602306"
Sep 15 10:36:46.462666 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.462595    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6jck\" (UniqueName: \"kubernetes.io/projected/7fad425d-7299-4e20-a756-0328f364a333-kube-api-access-x6jck\") pod \"test-curl-7879-602306\" (UID: \"7fad425d-7299-4e20-a756-0328f364a333\") " pod="opendatahub/test-curl-7879-602306"
Sep 15 10:36:46.522564 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.522531    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-602306"
Sep 15 10:36:46.791302 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.791278    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-602306"]
Sep 15 10:36:46.793441 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:36:46.793409    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fad425d_7299_4e20_a756_0328f364a333.slice/crio-a2519b5d0b7612b290002d8eb0869662e3852e88f59dbb62f6e33cda8fa582f4 WatchSource:0}: Error finding container a2519b5d0b7612b290002d8eb0869662e3852e88f59dbb62f6e33cda8fa582f4: Status 404 returned error can't find the container with id a2519b5d0b7612b290002d8eb0869662e3852e88f59dbb62f6e33cda8fa582f4
Sep 15 10:36:46.887166 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.887138    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-602306" event={"ID":"7fad425d-7299-4e20-a756-0328f364a333","Type":"ContainerStarted","Data":"c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0"}
Sep 15 10:36:46.887275 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.887180    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-602306" event={"ID":"7fad425d-7299-4e20-a756-0328f364a333","Type":"ContainerStarted","Data":"a2519b5d0b7612b290002d8eb0869662e3852e88f59dbb62f6e33cda8fa582f4"}
Sep 15 10:36:46.902139 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:46.902092    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7879-602306" podStartSLOduration=0.902077412 podStartE2EDuration="902.077412ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:36:46.900274791 +0000 UTC m=+1125.594430470" watchObservedRunningTime="2026-09-15 10:36:46.902077412 +0000 UTC m=+1125.596233091"
Sep 15 10:36:47.480442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:47.480355    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7879-602306"]
Sep 15 10:36:47.485174 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:47.485142    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7879-602306"]
Sep 15 10:36:47.892307 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:36:47.892272    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7879-602306" container="test-curl-7879-602306"
Sep 15 10:36:48.897179 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:48.897111    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7879-602306" podUID="7fad425d-7299-4e20-a756-0328f364a333" containerName="test-curl-7879-602306" containerID="cri-o://c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0" gracePeriod=2
Sep 15 10:36:51.085267 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:51.085244    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-602306"
Sep 15 10:36:51.098389 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:51.098364    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6jck\" (UniqueName: \"kubernetes.io/projected/7fad425d-7299-4e20-a756-0328f364a333-kube-api-access-x6jck\") pod \"7fad425d-7299-4e20-a756-0328f364a333\" (UID: \"7fad425d-7299-4e20-a756-0328f364a333\") "
Sep 15 10:36:51.100285 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:51.100260    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fad425d-7299-4e20-a756-0328f364a333-kube-api-access-x6jck" (OuterVolumeSpecName: "kube-api-access-x6jck") pod "7fad425d-7299-4e20-a756-0328f364a333" (UID: "7fad425d-7299-4e20-a756-0328f364a333"). InnerVolumeSpecName "kube-api-access-x6jck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:36:51.200289 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:51.200213    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6jck\" (UniqueName: \"kubernetes.io/projected/7fad425d-7299-4e20-a756-0328f364a333-kube-api-access-x6jck\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:36:51.786551 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:51.786518    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fad425d-7299-4e20-a756-0328f364a333" path="/var/lib/kubelet/pods/7fad425d-7299-4e20-a756-0328f364a333/volumes"
Sep 15 10:36:51.913466 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:51.913429    2584 generic.go:356] "Generic (PLEG): container finished" podID="7fad425d-7299-4e20-a756-0328f364a333" containerID="c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0" exitCode=137
Sep 15 10:36:51.913667 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:51.913481    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-602306"
Sep 15 10:36:51.913667 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:51.913510    2584 scope.go:117] "RemoveContainer" containerID="c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0"
Sep 15 10:36:51.922049 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:51.922028    2584 scope.go:117] "RemoveContainer" containerID="c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0"
Sep 15 10:36:51.922304 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:36:51.922287    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0\": container with ID starting with c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0 not found: ID does not exist" containerID="c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0"
Sep 15 10:36:51.922383 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:51.922310    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0"} err="failed to get container status \"c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0\": rpc error: code = NotFound desc = could not find container \"c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0\": container with ID starting with c5a6a7d257dbfaef2f76d97f7331b85f85e8d5be89ff4a70362f94bcfb08cbb0 not found: ID does not exist"
Sep 15 10:36:53.160612 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:53.160573    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:36:56.391534 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.391501    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"]
Sep 15 10:36:56.391948 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.391901    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fad425d-7299-4e20-a756-0328f364a333" containerName="test-curl-7879-602306"
Sep 15 10:36:56.391948 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.391913    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fad425d-7299-4e20-a756-0328f364a333" containerName="test-curl-7879-602306"
Sep 15 10:36:56.392020 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.391984    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fad425d-7299-4e20-a756-0328f364a333" containerName="test-curl-7879-602306"
Sep 15 10:36:56.396282 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.396262    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:36:56.398437 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.398413    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6111ecc0\""
Sep 15 10:36:56.399084 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.399064    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6111ecc0-dockercfg-vmn2w\""
Sep 15 10:36:56.405063 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.405043    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"]
Sep 15 10:36:56.453267 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.453232    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cf880f-e9a2-438f-a867-8a1e8e93b25a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk\" (UID: \"85cf880f-e9a2-438f-a867-8a1e8e93b25a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:36:56.453426 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.453296    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rglh9\" (UniqueName: \"kubernetes.io/projected/85cf880f-e9a2-438f-a867-8a1e8e93b25a-kube-api-access-rglh9\") pod \"payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk\" (UID: \"85cf880f-e9a2-438f-a867-8a1e8e93b25a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:36:56.553927 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.553894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cf880f-e9a2-438f-a867-8a1e8e93b25a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk\" (UID: \"85cf880f-e9a2-438f-a867-8a1e8e93b25a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:36:56.554086 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.553975    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rglh9\" (UniqueName: \"kubernetes.io/projected/85cf880f-e9a2-438f-a867-8a1e8e93b25a-kube-api-access-rglh9\") pod \"payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk\" (UID: \"85cf880f-e9a2-438f-a867-8a1e8e93b25a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:36:56.554592 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.554572    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cf880f-e9a2-438f-a867-8a1e8e93b25a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk\" (UID: \"85cf880f-e9a2-438f-a867-8a1e8e93b25a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:36:56.563970 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.563936    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rglh9\" (UniqueName: \"kubernetes.io/projected/85cf880f-e9a2-438f-a867-8a1e8e93b25a-kube-api-access-rglh9\") pod \"payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk\" (UID: \"85cf880f-e9a2-438f-a867-8a1e8e93b25a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:36:56.707497 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.707394    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:36:56.981326 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:56.981302    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"]
Sep 15 10:36:56.984055 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:36:56.984031    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cf880f_e9a2_438f_a867_8a1e8e93b25a.slice/crio-92f7fc3a75b83a2cca44798e1fb18a19682429504bb4d9b03f1fb623e25d9265 WatchSource:0}: Error finding container 92f7fc3a75b83a2cca44798e1fb18a19682429504bb4d9b03f1fb623e25d9265: Status 404 returned error can't find the container with id 92f7fc3a75b83a2cca44798e1fb18a19682429504bb4d9b03f1fb623e25d9265
Sep 15 10:36:57.945100 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:57.945064    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk" event={"ID":"85cf880f-e9a2-438f-a867-8a1e8e93b25a","Type":"ContainerStarted","Data":"7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727"}
Sep 15 10:36:57.945571 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:57.945155    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk" event={"ID":"85cf880f-e9a2-438f-a867-8a1e8e93b25a","Type":"ContainerStarted","Data":"92f7fc3a75b83a2cca44798e1fb18a19682429504bb4d9b03f1fb623e25d9265"}
Sep 15 10:36:57.945571 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:57.945213    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:36:57.963128 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:57.963080    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk" podStartSLOduration=1.963067584 podStartE2EDuration="1.963067584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:36:57.962136463 +0000 UTC m=+1136.656292146" watchObservedRunningTime="2026-09-15 10:36:57.963067584 +0000 UTC m=+1136.657223263"
Sep 15 10:36:59.364482 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.364425    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb" podUID="f41e216d-f2db-4324-bcf9-e2b13b17edb7" containerName="main" containerID="cri-o://ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6" gracePeriod=45
Sep 15 10:36:59.621480 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.621423    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:36:59.684721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.684688    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-dshm\") pod \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") "
Sep 15 10:36:59.684945 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.684740    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tls-certs\") pod \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") "
Sep 15 10:36:59.684945 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.684762    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-home\") pod \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") "
Sep 15 10:36:59.684945 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.684815    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m9fh\" (UniqueName: \"kubernetes.io/projected/f41e216d-f2db-4324-bcf9-e2b13b17edb7-kube-api-access-5m9fh\") pod \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") "
Sep 15 10:36:59.684945 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.684874    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-model-cache\") pod \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") "
Sep 15 10:36:59.684945 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.684938    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tmp-dir\") pod \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\" (UID: \"f41e216d-f2db-4324-bcf9-e2b13b17edb7\") "
Sep 15 10:36:59.685229 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.685119    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-home" (OuterVolumeSpecName: "home") pod "f41e216d-f2db-4324-bcf9-e2b13b17edb7" (UID: "f41e216d-f2db-4324-bcf9-e2b13b17edb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:36:59.685284 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.685256    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f41e216d-f2db-4324-bcf9-e2b13b17edb7" (UID: "f41e216d-f2db-4324-bcf9-e2b13b17edb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:36:59.685346 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.685266    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f41e216d-f2db-4324-bcf9-e2b13b17edb7" (UID: "f41e216d-f2db-4324-bcf9-e2b13b17edb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:36:59.685346 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.685315    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:36:59.687057 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.687023    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-dshm" (OuterVolumeSpecName: "dshm") pod "f41e216d-f2db-4324-bcf9-e2b13b17edb7" (UID: "f41e216d-f2db-4324-bcf9-e2b13b17edb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:36:59.687057 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.687023    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f41e216d-f2db-4324-bcf9-e2b13b17edb7" (UID: "f41e216d-f2db-4324-bcf9-e2b13b17edb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:36:59.687205 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.687120    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41e216d-f2db-4324-bcf9-e2b13b17edb7-kube-api-access-5m9fh" (OuterVolumeSpecName: "kube-api-access-5m9fh") pod "f41e216d-f2db-4324-bcf9-e2b13b17edb7" (UID: "f41e216d-f2db-4324-bcf9-e2b13b17edb7"). InnerVolumeSpecName "kube-api-access-5m9fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:36:59.786571 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.786535    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:36:59.786571 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.786571    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m9fh\" (UniqueName: \"kubernetes.io/projected/f41e216d-f2db-4324-bcf9-e2b13b17edb7-kube-api-access-5m9fh\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:36:59.786762 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.786587    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:36:59.786762 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.786600    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:36:59.786762 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.786612    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41e216d-f2db-4324-bcf9-e2b13b17edb7-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:36:59.956471 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.956386    2584 generic.go:356] "Generic (PLEG): container finished" podID="f41e216d-f2db-4324-bcf9-e2b13b17edb7" containerID="ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6" exitCode=0
Sep 15 10:36:59.956471 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.956425    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb" event={"ID":"f41e216d-f2db-4324-bcf9-e2b13b17edb7","Type":"ContainerDied","Data":"ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6"}
Sep 15 10:36:59.956471 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.956430    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"
Sep 15 10:36:59.956471 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.956455    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb" event={"ID":"f41e216d-f2db-4324-bcf9-e2b13b17edb7","Type":"ContainerDied","Data":"cee5f767fae9d8359ada60044ea7135ef8ae3678717eae10d079678af1c76860"}
Sep 15 10:36:59.956471 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.956470    2584 scope.go:117] "RemoveContainer" containerID="ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6"
Sep 15 10:36:59.965376 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.965318    2584 scope.go:117] "RemoveContainer" containerID="ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6"
Sep 15 10:36:59.965614 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:36:59.965591    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6\": container with ID starting with ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6 not found: ID does not exist" containerID="ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6"
Sep 15 10:36:59.965677 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.965626    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6"} err="failed to get container status \"ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6\": rpc error: code = NotFound desc = could not find container \"ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6\": container with ID starting with ab95bf72bfa2b5eb583d49fe1a389ce962306d2d27d270aba29c36eef83c91c6 not found: ID does not exist"
Sep 15 10:36:59.971460 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.971435    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"]
Sep 15 10:36:59.973197 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:36:59.973180    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-aeb4e3/explicit-model-aeb4e3-kserve-7cf49db797-qsrwb"]
Sep 15 10:37:01.438631 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.438590    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"]
Sep 15 10:37:01.439232 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.439211    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f41e216d-f2db-4324-bcf9-e2b13b17edb7" containerName="main"
Sep 15 10:37:01.439316 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.439234    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f41e216d-f2db-4324-bcf9-e2b13b17edb7" containerName="main"
Sep 15 10:37:01.439374 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.439347    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f41e216d-f2db-4324-bcf9-e2b13b17edb7" containerName="main"
Sep 15 10:37:01.443608 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.443588    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.446233 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.446179    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ff0655\"/\"e2e-conflict-a-55001805-kserve-self-signed-certs\""
Sep 15 10:37:01.450956 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.450931    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"]
Sep 15 10:37:01.504082 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.504041    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d067108e-9ede-4f74-b36f-6c0237057c1b-tls-certs\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.504234 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.504089    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf999\" (UniqueName: \"kubernetes.io/projected/d067108e-9ede-4f74-b36f-6c0237057c1b-kube-api-access-gf999\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.504234 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.504118    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-model-cache\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.504234 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.504136    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-dshm\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.504234 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.504168    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-home\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.504234 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.504192    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-tmp-dir\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.604773 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.604737    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d067108e-9ede-4f74-b36f-6c0237057c1b-tls-certs\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.604962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.604783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf999\" (UniqueName: \"kubernetes.io/projected/d067108e-9ede-4f74-b36f-6c0237057c1b-kube-api-access-gf999\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.604962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.604832    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-model-cache\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.605092 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.604984    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-dshm\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.605092 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.605062    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-home\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.605209 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.605101    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-tmp-dir\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.605209 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.605180    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-model-cache\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.605353 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.605332    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-home\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.605469 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.605450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-tmp-dir\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.607046 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.607023    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-dshm\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.607217 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.607202    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d067108e-9ede-4f74-b36f-6c0237057c1b-tls-certs\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.611885 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.611862    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf999\" (UniqueName: \"kubernetes.io/projected/d067108e-9ede-4f74-b36f-6c0237057c1b-kube-api-access-gf999\") pod \"e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.761844 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.761812    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:01.786729 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:01.786696    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f41e216d-f2db-4324-bcf9-e2b13b17edb7" path="/var/lib/kubelet/pods/f41e216d-f2db-4324-bcf9-e2b13b17edb7/volumes"
Sep 15 10:37:02.034291 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:02.034267    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"]
Sep 15 10:37:02.036741 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:37:02.036712    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd067108e_9ede_4f74_b36f_6c0237057c1b.slice/crio-f4f4c96ab9a88c1f3391ec67eccce60582dfd89f9566148e8028790771770fbb WatchSource:0}: Error finding container f4f4c96ab9a88c1f3391ec67eccce60582dfd89f9566148e8028790771770fbb: Status 404 returned error can't find the container with id f4f4c96ab9a88c1f3391ec67eccce60582dfd89f9566148e8028790771770fbb
Sep 15 10:37:02.974851 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:02.974793    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg" event={"ID":"d067108e-9ede-4f74-b36f-6c0237057c1b","Type":"ContainerStarted","Data":"6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d"}
Sep 15 10:37:02.974851 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:02.974859    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg" event={"ID":"d067108e-9ede-4f74-b36f-6c0237057c1b","Type":"ContainerStarted","Data":"f4f4c96ab9a88c1f3391ec67eccce60582dfd89f9566148e8028790771770fbb"}
Sep 15 10:37:02.992550 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:02.992495    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg" podStartSLOduration=1.992478648 podStartE2EDuration="1.992478648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:37:02.99118319 +0000 UTC m=+1141.685338870" watchObservedRunningTime="2026-09-15 10:37:02.992478648 +0000 UTC m=+1141.686634327"
Sep 15 10:37:08.952396 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:08.952367    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:37:11.761930 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:11.761892    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:11.761930 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:11.761939    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:11.775045 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:11.775016    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:12.031159 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:12.031076    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:37:22.670651 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:22.670602    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:37:45.349431 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.349396    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"]
Sep 15 10:37:45.359182 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.359159    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"]
Sep 15 10:37:45.359309 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.359242    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.361575 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.361549    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8c1047\"/\"distinct-w0-8c1047-kserve-self-signed-certs\""
Sep 15 10:37:45.442699 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.442664    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89zq7\" (UniqueName: \"kubernetes.io/projected/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-kube-api-access-89zq7\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.442897 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.442713    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tls-certs\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.442897 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.442848    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-home\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.442897 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.442879    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tmp-dir\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.443021 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.442901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-dshm\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.443021 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.443003    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-model-cache\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.544043 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.544005    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89zq7\" (UniqueName: \"kubernetes.io/projected/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-kube-api-access-89zq7\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.544215 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.544056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tls-certs\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.544215 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.544113    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-home\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.544215 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.544129    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tmp-dir\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.544215 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.544148    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-dshm\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.544215 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.544212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-model-cache\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.544577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.544553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-home\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.544652 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.544592    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tmp-dir\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.544652 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.544607    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-model-cache\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.546418 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.546393    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-dshm\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.546686 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.546669    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tls-certs\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.551685 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.551657    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89zq7\" (UniqueName: \"kubernetes.io/projected/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-kube-api-access-89zq7\") pod \"distinct-w0-8c1047-kserve-5975f79c8b-9b4mn\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") " pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.670646 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.670558    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:45.939104 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:45.938970    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"]
Sep 15 10:37:45.942007 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:37:45.941982    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd40ebc9b_bf8f_49fb_b6ed_95fa452f7aa0.slice/crio-f7a903fbb0c27cffff7dc0c33a912f6f1f452793f5518c953968f90cc644414e WatchSource:0}: Error finding container f7a903fbb0c27cffff7dc0c33a912f6f1f452793f5518c953968f90cc644414e: Status 404 returned error can't find the container with id f7a903fbb0c27cffff7dc0c33a912f6f1f452793f5518c953968f90cc644414e
Sep 15 10:37:46.192980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:46.192890    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn" event={"ID":"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0","Type":"ContainerStarted","Data":"6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224"}
Sep 15 10:37:46.192980 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:46.192931    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn" event={"ID":"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0","Type":"ContainerStarted","Data":"f7a903fbb0c27cffff7dc0c33a912f6f1f452793f5518c953968f90cc644414e"}
Sep 15 10:37:46.208310 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:46.208254    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn" podStartSLOduration=1.208237346 podStartE2EDuration="1.208237346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:37:46.207638479 +0000 UTC m=+1184.901794160" watchObservedRunningTime="2026-09-15 10:37:46.208237346 +0000 UTC m=+1184.902393029"
Sep 15 10:37:55.670853 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:55.670793    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:55.671336 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:55.670868    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:55.683594 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:55.683566    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:56.259231 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:56.259197    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:37:56.272145 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:37:56.272113    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:38:01.852486 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:01.852452    2584 scope.go:117] "RemoveContainer" containerID="f9bfa384ce9666e99c98d58352eb2de47aaba9002e68a84bcb0a6bb433aaf569"
Sep 15 10:38:19.755973 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:19.755932    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:38:35.333550 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:35.333516    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"]
Sep 15 10:38:39.483346 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:39.483312    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"]
Sep 15 10:38:39.483874 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:39.483606    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb" podUID="de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223" containerName="payload-processing" containerID="cri-o://8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076" gracePeriod=30
Sep 15 10:38:39.727625 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:39.727600    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:38:39.772871 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:39.772839    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-plugins-config-volume\") pod \"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223\" (UID: \"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223\") "
Sep 15 10:38:39.772871 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:39.772872    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl5g2\" (UniqueName: \"kubernetes.io/projected/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-kube-api-access-sl5g2\") pod \"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223\" (UID: \"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223\") "
Sep 15 10:38:39.773270 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:39.773241    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223" (UID: "de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:38:39.775028 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:39.774998    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-kube-api-access-sl5g2" (OuterVolumeSpecName: "kube-api-access-sl5g2") pod "de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223" (UID: "de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223"). InnerVolumeSpecName "kube-api-access-sl5g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:38:39.874057 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:39.874016    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:38:39.874057 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:39.874047    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl5g2\" (UniqueName: \"kubernetes.io/projected/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223-kube-api-access-sl5g2\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:38:40.459448 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:40.459413    2584 generic.go:356] "Generic (PLEG): container finished" podID="de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223" containerID="8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076" exitCode=0
Sep 15 10:38:40.459629 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:40.459457    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"
Sep 15 10:38:40.459629 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:40.459481    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb" event={"ID":"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223","Type":"ContainerDied","Data":"8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076"}
Sep 15 10:38:40.459629 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:40.459522    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb" event={"ID":"de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223","Type":"ContainerDied","Data":"e8f26109a0413b5c1c4f45bd747713dca1a2ce1074a7750ccd9cfcddf65eb980"}
Sep 15 10:38:40.459629 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:40.459543    2584 scope.go:117] "RemoveContainer" containerID="8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076"
Sep 15 10:38:40.468577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:40.468557    2584 scope.go:117] "RemoveContainer" containerID="8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076"
Sep 15 10:38:40.468889 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:38:40.468864    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076\": container with ID starting with 8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076 not found: ID does not exist" containerID="8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076"
Sep 15 10:38:40.468985 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:40.468893    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076"} err="failed to get container status \"8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076\": rpc error: code = NotFound desc = could not find container \"8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076\": container with ID starting with 8a9fc56a7b039a6fe1af7377adf9a375021694e4fd5248d7999ba594d4bad076 not found: ID does not exist"
Sep 15 10:38:40.476279 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:40.476190    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"]
Sep 15 10:38:40.478420 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:40.478398    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-86a8f5-7896548df7-ljgcb"]
Sep 15 10:38:41.785069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:41.785028    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223" path="/var/lib/kubelet/pods/de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223/volumes"
Sep 15 10:38:47.765457 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:47.765424    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:38:50.353680 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.353629    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf" podUID="09d85a8a-137a-48d1-9610-2066ddad122a" containerName="main" containerID="cri-o://ca5e615ca01c87e6c636c81d053473c52078e7349a611f248e5702bf51d5d534" gracePeriod=45
Sep 15 10:38:50.512362 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.512331    2584 generic.go:356] "Generic (PLEG): container finished" podID="09d85a8a-137a-48d1-9610-2066ddad122a" containerID="ca5e615ca01c87e6c636c81d053473c52078e7349a611f248e5702bf51d5d534" exitCode=0
Sep 15 10:38:50.512536 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.512399    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf" event={"ID":"09d85a8a-137a-48d1-9610-2066ddad122a","Type":"ContainerDied","Data":"ca5e615ca01c87e6c636c81d053473c52078e7349a611f248e5702bf51d5d534"}
Sep 15 10:38:50.607589 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.607531    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:38:50.677922 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.677884    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrc5r\" (UniqueName: \"kubernetes.io/projected/09d85a8a-137a-48d1-9610-2066ddad122a-kube-api-access-lrc5r\") pod \"09d85a8a-137a-48d1-9610-2066ddad122a\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") "
Sep 15 10:38:50.678087 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.677950    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-home\") pod \"09d85a8a-137a-48d1-9610-2066ddad122a\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") "
Sep 15 10:38:50.678087 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.677977    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-tmp-dir\") pod \"09d85a8a-137a-48d1-9610-2066ddad122a\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") "
Sep 15 10:38:50.678087 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.678040    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d85a8a-137a-48d1-9610-2066ddad122a-tls-certs\") pod \"09d85a8a-137a-48d1-9610-2066ddad122a\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") "
Sep 15 10:38:50.678087 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.678069    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-model-cache\") pod \"09d85a8a-137a-48d1-9610-2066ddad122a\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") "
Sep 15 10:38:50.678341 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.678099    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-dshm\") pod \"09d85a8a-137a-48d1-9610-2066ddad122a\" (UID: \"09d85a8a-137a-48d1-9610-2066ddad122a\") "
Sep 15 10:38:50.678341 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.678277    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09d85a8a-137a-48d1-9610-2066ddad122a" (UID: "09d85a8a-137a-48d1-9610-2066ddad122a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:38:50.678448 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.678337    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-home" (OuterVolumeSpecName: "home") pod "09d85a8a-137a-48d1-9610-2066ddad122a" (UID: "09d85a8a-137a-48d1-9610-2066ddad122a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:38:50.678448 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.678366    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-model-cache" (OuterVolumeSpecName: "model-cache") pod "09d85a8a-137a-48d1-9610-2066ddad122a" (UID: "09d85a8a-137a-48d1-9610-2066ddad122a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:38:50.678654 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.678633    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:38:50.678654 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.678653    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:38:50.678891 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.678662    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:38:50.680253 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.680226    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d85a8a-137a-48d1-9610-2066ddad122a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09d85a8a-137a-48d1-9610-2066ddad122a" (UID: "09d85a8a-137a-48d1-9610-2066ddad122a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:38:50.680253 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.680238    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-dshm" (OuterVolumeSpecName: "dshm") pod "09d85a8a-137a-48d1-9610-2066ddad122a" (UID: "09d85a8a-137a-48d1-9610-2066ddad122a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:38:50.680374 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.680343    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d85a8a-137a-48d1-9610-2066ddad122a-kube-api-access-lrc5r" (OuterVolumeSpecName: "kube-api-access-lrc5r") pod "09d85a8a-137a-48d1-9610-2066ddad122a" (UID: "09d85a8a-137a-48d1-9610-2066ddad122a"). InnerVolumeSpecName "kube-api-access-lrc5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:38:50.779818 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.779776    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrc5r\" (UniqueName: \"kubernetes.io/projected/09d85a8a-137a-48d1-9610-2066ddad122a-kube-api-access-lrc5r\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:38:50.779991 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.779829    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d85a8a-137a-48d1-9610-2066ddad122a-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:38:50.779991 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:50.779839    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d85a8a-137a-48d1-9610-2066ddad122a-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:38:51.518737 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:51.518706    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"
Sep 15 10:38:51.518737 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:51.518727    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf" event={"ID":"09d85a8a-137a-48d1-9610-2066ddad122a","Type":"ContainerDied","Data":"e92b51c0e8c071e902cd399ea70693388fe8d660c79b6691c9a7fa62735bfa39"}
Sep 15 10:38:51.519354 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:51.518784    2584 scope.go:117] "RemoveContainer" containerID="ca5e615ca01c87e6c636c81d053473c52078e7349a611f248e5702bf51d5d534"
Sep 15 10:38:51.539192 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:51.539167    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"]
Sep 15 10:38:51.542767 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:51.542741    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-86a8f5/test-model-86a8f5-kserve-77bd97df8f-t4xsf"]
Sep 15 10:38:51.786517 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:51.786437    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d85a8a-137a-48d1-9610-2066ddad122a" path="/var/lib/kubelet/pods/09d85a8a-137a-48d1-9610-2066ddad122a/volumes"
Sep 15 10:38:59.070719 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.070643    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"]
Sep 15 10:38:59.071078 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.071043    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223" containerName="payload-processing"
Sep 15 10:38:59.071078 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.071055    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223" containerName="payload-processing"
Sep 15 10:38:59.071078 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.071065    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d85a8a-137a-48d1-9610-2066ddad122a" containerName="main"
Sep 15 10:38:59.071078 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.071072    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d85a8a-137a-48d1-9610-2066ddad122a" containerName="main"
Sep 15 10:38:59.071226 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.071132    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5bbc13-dd07-4bb9-94a7-7ca7b9eb2223" containerName="payload-processing"
Sep 15 10:38:59.071226 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.071141    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d85a8a-137a-48d1-9610-2066ddad122a" containerName="main"
Sep 15 10:38:59.075706 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.075689    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.079699 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.079664    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-948cb7\"/\"kube-root-ca.crt\""
Sep 15 10:38:59.079699 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.079686    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-948cb7\"/\"default-dockercfg-rckmr\""
Sep 15 10:38:59.079921 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.079702    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-948cb7\"/\"openshift-service-ca.crt\""
Sep 15 10:38:59.079921 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.079689    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-948cb7\"/\"noten-model-948cb7-kserve-self-signed-certs\""
Sep 15 10:38:59.084403 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.084385    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"]
Sep 15 10:38:59.162907 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.162867    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tmp-dir\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.162907 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.162914    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-home\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.163134 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.163037    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-dshm\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.163134 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.163116    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-model-cache\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.163215 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.163157    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k45p4\" (UniqueName: \"kubernetes.io/projected/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-kube-api-access-k45p4\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.163215 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.163208    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tls-certs\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.264657 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.264621    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k45p4\" (UniqueName: \"kubernetes.io/projected/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-kube-api-access-k45p4\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.264846 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.264676    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tls-certs\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.264846 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.264707    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tmp-dir\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.264846 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.264735    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-home\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.264846 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.264825    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-dshm\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.265034 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.264895    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-model-cache\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.265195 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.265172    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tmp-dir\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.265256 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.265190    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-home\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.265315 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.265252    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-model-cache\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.267068 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.267046    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-dshm\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.267237 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.267220    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tls-certs\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.271775 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.271726    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k45p4\" (UniqueName: \"kubernetes.io/projected/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-kube-api-access-k45p4\") pod \"noten-model-948cb7-kserve-6b59859755-rz7lr\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") " pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.386908 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.386822    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:38:59.670378 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:38:59.670346    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"]
Sep 15 10:38:59.672423 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:38:59.672391    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18fe73b5_d76f_490a_bcb6_a9bdb8a942d9.slice/crio-51bc54a6f860db57900094a2a5392ce5a3a0f28c9e03393c848fb75047982fe4 WatchSource:0}: Error finding container 51bc54a6f860db57900094a2a5392ce5a3a0f28c9e03393c848fb75047982fe4: Status 404 returned error can't find the container with id 51bc54a6f860db57900094a2a5392ce5a3a0f28c9e03393c848fb75047982fe4
Sep 15 10:39:00.573645 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:00.573608    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr" event={"ID":"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9","Type":"ContainerStarted","Data":"5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d"}
Sep 15 10:39:00.573645 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:00.573649    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr" event={"ID":"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9","Type":"ContainerStarted","Data":"51bc54a6f860db57900094a2a5392ce5a3a0f28c9e03393c848fb75047982fe4"}
Sep 15 10:39:00.590295 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:00.590243    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr" podStartSLOduration=1.590228335 podStartE2EDuration="1.590228335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:39:00.588561022 +0000 UTC m=+1259.282716703" watchObservedRunningTime="2026-09-15 10:39:00.590228335 +0000 UTC m=+1259.284384016"
Sep 15 10:39:09.387871 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:09.387836    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:39:09.388283 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:09.387887    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:39:09.400565 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:09.400537    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:39:09.628654 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:09.628619    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:39:10.396515 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.396482    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"]
Sep 15 10:39:10.396984 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.396717    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf" podUID="38ae53bd-2579-40a0-8100-ac32d8f2d600" containerName="payload-processing" containerID="cri-o://8786ea15d7b61a934fadcd7169713ffb5b99c1f07a29533279bd50fcae84973f" gracePeriod=30
Sep 15 10:39:10.404034 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.404006    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"]
Sep 15 10:39:10.404270 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.404223    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv" podUID="e4c0f1f5-1dd4-4878-9a90-c97579113319" containerName="payload-pre-processing" containerID="cri-o://83c8e3e9b3b45ecf2add8a7f430776fb45b0df38b4d6aabf6616836f94c57045" gracePeriod=30
Sep 15 10:39:10.624780 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.624752    2584 generic.go:356] "Generic (PLEG): container finished" podID="38ae53bd-2579-40a0-8100-ac32d8f2d600" containerID="8786ea15d7b61a934fadcd7169713ffb5b99c1f07a29533279bd50fcae84973f" exitCode=0
Sep 15 10:39:10.624992 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.624854    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf" event={"ID":"38ae53bd-2579-40a0-8100-ac32d8f2d600","Type":"ContainerDied","Data":"8786ea15d7b61a934fadcd7169713ffb5b99c1f07a29533279bd50fcae84973f"}
Sep 15 10:39:10.626920 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.626840    2584 generic.go:356] "Generic (PLEG): container finished" podID="e4c0f1f5-1dd4-4878-9a90-c97579113319" containerID="83c8e3e9b3b45ecf2add8a7f430776fb45b0df38b4d6aabf6616836f94c57045" exitCode=0
Sep 15 10:39:10.627148 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.627126    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv" event={"ID":"e4c0f1f5-1dd4-4878-9a90-c97579113319","Type":"ContainerDied","Data":"83c8e3e9b3b45ecf2add8a7f430776fb45b0df38b4d6aabf6616836f94c57045"}
Sep 15 10:39:10.686462 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.686440    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:39:10.689135 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.689114    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:39:10.777454 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.777417    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c0f1f5-1dd4-4878-9a90-c97579113319-plugins-config-volume\") pod \"e4c0f1f5-1dd4-4878-9a90-c97579113319\" (UID: \"e4c0f1f5-1dd4-4878-9a90-c97579113319\") "
Sep 15 10:39:10.777617 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.777500    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8swn\" (UniqueName: \"kubernetes.io/projected/e4c0f1f5-1dd4-4878-9a90-c97579113319-kube-api-access-r8swn\") pod \"e4c0f1f5-1dd4-4878-9a90-c97579113319\" (UID: \"e4c0f1f5-1dd4-4878-9a90-c97579113319\") "
Sep 15 10:39:10.777617 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.777528    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ae53bd-2579-40a0-8100-ac32d8f2d600-plugins-config-volume\") pod \"38ae53bd-2579-40a0-8100-ac32d8f2d600\" (UID: \"38ae53bd-2579-40a0-8100-ac32d8f2d600\") "
Sep 15 10:39:10.777617 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.777594    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgmjc\" (UniqueName: \"kubernetes.io/projected/38ae53bd-2579-40a0-8100-ac32d8f2d600-kube-api-access-jgmjc\") pod \"38ae53bd-2579-40a0-8100-ac32d8f2d600\" (UID: \"38ae53bd-2579-40a0-8100-ac32d8f2d600\") "
Sep 15 10:39:10.777846 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.777790    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c0f1f5-1dd4-4878-9a90-c97579113319-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4c0f1f5-1dd4-4878-9a90-c97579113319" (UID: "e4c0f1f5-1dd4-4878-9a90-c97579113319"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:39:10.777976 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.777936    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ae53bd-2579-40a0-8100-ac32d8f2d600-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38ae53bd-2579-40a0-8100-ac32d8f2d600" (UID: "38ae53bd-2579-40a0-8100-ac32d8f2d600"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:39:10.778042 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.778007    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c0f1f5-1dd4-4878-9a90-c97579113319-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:39:10.779637 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.779613    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ae53bd-2579-40a0-8100-ac32d8f2d600-kube-api-access-jgmjc" (OuterVolumeSpecName: "kube-api-access-jgmjc") pod "38ae53bd-2579-40a0-8100-ac32d8f2d600" (UID: "38ae53bd-2579-40a0-8100-ac32d8f2d600"). InnerVolumeSpecName "kube-api-access-jgmjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:39:10.779790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.779773    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c0f1f5-1dd4-4878-9a90-c97579113319-kube-api-access-r8swn" (OuterVolumeSpecName: "kube-api-access-r8swn") pod "e4c0f1f5-1dd4-4878-9a90-c97579113319" (UID: "e4c0f1f5-1dd4-4878-9a90-c97579113319"). InnerVolumeSpecName "kube-api-access-r8swn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:39:10.879380 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.879344    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8swn\" (UniqueName: \"kubernetes.io/projected/e4c0f1f5-1dd4-4878-9a90-c97579113319-kube-api-access-r8swn\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:39:10.879380 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.879369    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ae53bd-2579-40a0-8100-ac32d8f2d600-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:39:10.879380 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:10.879383    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgmjc\" (UniqueName: \"kubernetes.io/projected/38ae53bd-2579-40a0-8100-ac32d8f2d600-kube-api-access-jgmjc\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:39:11.633849 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.633818    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"
Sep 15 10:39:11.633849 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.633832    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf" event={"ID":"38ae53bd-2579-40a0-8100-ac32d8f2d600","Type":"ContainerDied","Data":"66be11792ad528838575edf88426987b1db163f97439d9b895df3ccfc33358a3"}
Sep 15 10:39:11.634376 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.633899    2584 scope.go:117] "RemoveContainer" containerID="8786ea15d7b61a934fadcd7169713ffb5b99c1f07a29533279bd50fcae84973f"
Sep 15 10:39:11.635238 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.635224    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"
Sep 15 10:39:11.635307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.635267    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv" event={"ID":"e4c0f1f5-1dd4-4878-9a90-c97579113319","Type":"ContainerDied","Data":"68234d197502d0f8d31d6a9ddf421ad062e516fa668ab8e1a74c59be97ef7f28"}
Sep 15 10:39:11.644956 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.644934    2584 scope.go:117] "RemoveContainer" containerID="83c8e3e9b3b45ecf2add8a7f430776fb45b0df38b4d6aabf6616836f94c57045"
Sep 15 10:39:11.657399 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.657364    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"]
Sep 15 10:39:11.663105 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.663082    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-234840-576cdb7845-xrrcf"]
Sep 15 10:39:11.673751 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.673718    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"]
Sep 15 10:39:11.677115 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.677095    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-234840-5bc7895c86-hvxgv"]
Sep 15 10:39:11.785963 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.785936    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ae53bd-2579-40a0-8100-ac32d8f2d600" path="/var/lib/kubelet/pods/38ae53bd-2579-40a0-8100-ac32d8f2d600/volumes"
Sep 15 10:39:11.786330 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:11.786317    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c0f1f5-1dd4-4878-9a90-c97579113319" path="/var/lib/kubelet/pods/e4c0f1f5-1dd4-4878-9a90-c97579113319/volumes"
Sep 15 10:39:17.658523 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.658485    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"]
Sep 15 10:39:17.659101 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.659077    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ae53bd-2579-40a0-8100-ac32d8f2d600" containerName="payload-processing"
Sep 15 10:39:17.659150 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.659106    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ae53bd-2579-40a0-8100-ac32d8f2d600" containerName="payload-processing"
Sep 15 10:39:17.659150 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.659122    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c0f1f5-1dd4-4878-9a90-c97579113319" containerName="payload-pre-processing"
Sep 15 10:39:17.659150 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.659131    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c0f1f5-1dd4-4878-9a90-c97579113319" containerName="payload-pre-processing"
Sep 15 10:39:17.659257 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.659245    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ae53bd-2579-40a0-8100-ac32d8f2d600" containerName="payload-processing"
Sep 15 10:39:17.659293 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.659261    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c0f1f5-1dd4-4878-9a90-c97579113319" containerName="payload-pre-processing"
Sep 15 10:39:17.664497 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.664476    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.667828 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.667402    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ff0655\"/\"e2e-conflict-b-55001805-kserve-self-signed-certs\""
Sep 15 10:39:17.669904 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.669046    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"]
Sep 15 10:39:17.751917 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.751876    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2kn\" (UniqueName: \"kubernetes.io/projected/e696aa28-7767-43f8-b8d9-61765f8436f6-kube-api-access-kp2kn\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.752108 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.751939    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e696aa28-7767-43f8-b8d9-61765f8436f6-tls-certs\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.752108 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.751993    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-model-cache\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.752108 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.752090    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-home\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.752297 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.752133    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-dshm\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.752297 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.752163    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-tmp-dir\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.852728 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.852684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-home\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.852728 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.852734    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-dshm\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.853059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.852766    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-tmp-dir\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.853059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.852892    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2kn\" (UniqueName: \"kubernetes.io/projected/e696aa28-7767-43f8-b8d9-61765f8436f6-kube-api-access-kp2kn\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.853059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.852967    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e696aa28-7767-43f8-b8d9-61765f8436f6-tls-certs\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.853059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.853007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-model-cache\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.853204 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.853157    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-home\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.853544 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.853520    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-tmp-dir\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.853687 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.853669    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-model-cache\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.855419 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.855397    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-dshm\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.855672 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.855653    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e696aa28-7767-43f8-b8d9-61765f8436f6-tls-certs\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.860625 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.860603    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2kn\" (UniqueName: \"kubernetes.io/projected/e696aa28-7767-43f8-b8d9-61765f8436f6-kube-api-access-kp2kn\") pod \"e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") " pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:17.980907 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:17.980824    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:18.251431 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:18.251402    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"]
Sep 15 10:39:18.253463 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:39:18.253432    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode696aa28_7767_43f8_b8d9_61765f8436f6.slice/crio-023e41d3577eea6f0ff379444a949a374047685c06efb11ba5098ec1565e7688 WatchSource:0}: Error finding container 023e41d3577eea6f0ff379444a949a374047685c06efb11ba5098ec1565e7688: Status 404 returned error can't find the container with id 023e41d3577eea6f0ff379444a949a374047685c06efb11ba5098ec1565e7688
Sep 15 10:39:18.677926 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:18.677831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl" event={"ID":"e696aa28-7767-43f8-b8d9-61765f8436f6","Type":"ContainerStarted","Data":"0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22"}
Sep 15 10:39:18.677926 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:18.677877    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl" event={"ID":"e696aa28-7767-43f8-b8d9-61765f8436f6","Type":"ContainerStarted","Data":"023e41d3577eea6f0ff379444a949a374047685c06efb11ba5098ec1565e7688"}
Sep 15 10:39:18.696577 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:18.696527    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl" podStartSLOduration=1.696511068 podStartE2EDuration="1.696511068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:39:18.693973067 +0000 UTC m=+1277.388128747" watchObservedRunningTime="2026-09-15 10:39:18.696511068 +0000 UTC m=+1277.390666773"
Sep 15 10:39:27.981404 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:27.981366    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:27.981954 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:27.981417    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:27.993844 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:27.993820    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:28.739411 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:28.739377    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:39:30.947041 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:30.947009    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"]
Sep 15 10:39:30.950465 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:30.950449    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:39:30.952829 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:30.952792    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-83e814\""
Sep 15 10:39:30.952942 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:30.952856    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-83e814-dockercfg-mxcn4\""
Sep 15 10:39:30.961348 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:30.961324    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"]
Sep 15 10:39:31.086761 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.086719    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ab1f0a-494e-4c75-a991-03b236ceb215-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk\" (UID: \"78ab1f0a-494e-4c75-a991-03b236ceb215\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:39:31.086997 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.086869    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgsz\" (UniqueName: \"kubernetes.io/projected/78ab1f0a-494e-4c75-a991-03b236ceb215-kube-api-access-wcgsz\") pod \"payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk\" (UID: \"78ab1f0a-494e-4c75-a991-03b236ceb215\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:39:31.152442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.152397    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:39:31.188344 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.188307    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgsz\" (UniqueName: \"kubernetes.io/projected/78ab1f0a-494e-4c75-a991-03b236ceb215-kube-api-access-wcgsz\") pod \"payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk\" (UID: \"78ab1f0a-494e-4c75-a991-03b236ceb215\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:39:31.188511 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.188388    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ab1f0a-494e-4c75-a991-03b236ceb215-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk\" (UID: \"78ab1f0a-494e-4c75-a991-03b236ceb215\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:39:31.189114 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.189095    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ab1f0a-494e-4c75-a991-03b236ceb215-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk\" (UID: \"78ab1f0a-494e-4c75-a991-03b236ceb215\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:39:31.195379 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.195353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgsz\" (UniqueName: \"kubernetes.io/projected/78ab1f0a-494e-4c75-a991-03b236ceb215-kube-api-access-wcgsz\") pod \"payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk\" (UID: \"78ab1f0a-494e-4c75-a991-03b236ceb215\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:39:31.262885 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.262852    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:39:31.537686 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.537661    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"]
Sep 15 10:39:31.540700 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:39:31.540670    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ab1f0a_494e_4c75_a991_03b236ceb215.slice/crio-a53eaf2ca0b2c1e64cb906f0cfc177a7b2a89afac2374e1f7430a42805650fca WatchSource:0}: Error finding container a53eaf2ca0b2c1e64cb906f0cfc177a7b2a89afac2374e1f7430a42805650fca: Status 404 returned error can't find the container with id a53eaf2ca0b2c1e64cb906f0cfc177a7b2a89afac2374e1f7430a42805650fca
Sep 15 10:39:31.746687 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.746634    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk" event={"ID":"78ab1f0a-494e-4c75-a991-03b236ceb215","Type":"ContainerStarted","Data":"2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf"}
Sep 15 10:39:31.746902 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.746744    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk" event={"ID":"78ab1f0a-494e-4c75-a991-03b236ceb215","Type":"ContainerStarted","Data":"a53eaf2ca0b2c1e64cb906f0cfc177a7b2a89afac2374e1f7430a42805650fca"}
Sep 15 10:39:31.746902 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.746776    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:39:31.764795 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:31.764735    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk" podStartSLOduration=1.764719955 podStartE2EDuration="1.764719955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:39:31.762250064 +0000 UTC m=+1290.456405748" watchObservedRunningTime="2026-09-15 10:39:31.764719955 +0000 UTC m=+1290.458875634"
Sep 15 10:39:38.814379 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:38.814344    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"]
Sep 15 10:39:42.754321 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:42.754290    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:39:53.833977 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:53.833939    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr" podUID="18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" containerName="main" containerID="cri-o://5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d" gracePeriod=45
Sep 15 10:39:54.077820 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.077772    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:39:54.121273 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.121185    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-dshm\") pod \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") "
Sep 15 10:39:54.121273 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.121238    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k45p4\" (UniqueName: \"kubernetes.io/projected/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-kube-api-access-k45p4\") pod \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") "
Sep 15 10:39:54.121493 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.121280    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tmp-dir\") pod \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") "
Sep 15 10:39:54.121493 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.121317    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tls-certs\") pod \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") "
Sep 15 10:39:54.121493 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.121375    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-home\") pod \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") "
Sep 15 10:39:54.121493 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.121462    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-model-cache\") pod \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\" (UID: \"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9\") "
Sep 15 10:39:54.121700 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.121564    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" (UID: "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:39:54.121792 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.121771    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-home" (OuterVolumeSpecName: "home") pod "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" (UID: "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:39:54.121932 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.121914    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:39:54.122059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.122038    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:39:54.122059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.121962    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" (UID: "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:39:54.123760 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.123735    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-kube-api-access-k45p4" (OuterVolumeSpecName: "kube-api-access-k45p4") pod "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" (UID: "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9"). InnerVolumeSpecName "kube-api-access-k45p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:39:54.123868 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.123773    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-dshm" (OuterVolumeSpecName: "dshm") pod "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" (UID: "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:39:54.123868 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.123773    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" (UID: "18fe73b5-d76f-490a-bcb6-a9bdb8a942d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:39:54.223291 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.223258    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:39:54.223291 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.223287    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k45p4\" (UniqueName: \"kubernetes.io/projected/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-kube-api-access-k45p4\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:39:54.223291 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.223298    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:39:54.223538 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.223309    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:39:54.863868 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.863829    2584 generic.go:356] "Generic (PLEG): container finished" podID="18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" containerID="5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d" exitCode=0
Sep 15 10:39:54.863868 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.863870    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"
Sep 15 10:39:54.864354 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.863889    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr" event={"ID":"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9","Type":"ContainerDied","Data":"5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d"}
Sep 15 10:39:54.864354 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.863934    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr" event={"ID":"18fe73b5-d76f-490a-bcb6-a9bdb8a942d9","Type":"ContainerDied","Data":"51bc54a6f860db57900094a2a5392ce5a3a0f28c9e03393c848fb75047982fe4"}
Sep 15 10:39:54.864354 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.863952    2584 scope.go:117] "RemoveContainer" containerID="5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d"
Sep 15 10:39:54.874352 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.874335    2584 scope.go:117] "RemoveContainer" containerID="5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d"
Sep 15 10:39:54.874651 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:39:54.874634    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d\": container with ID starting with 5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d not found: ID does not exist" containerID="5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d"
Sep 15 10:39:54.874713 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.874663    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d"} err="failed to get container status \"5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d\": rpc error: code = NotFound desc = could not find container \"5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d\": container with ID starting with 5540b60bd4a537f98cae07f0a951c25306c51d0bca9e2546e7051766cdef759d not found: ID does not exist"
Sep 15 10:39:54.884872 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.884845    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"]
Sep 15 10:39:54.890145 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:54.890124    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-948cb7/noten-model-948cb7-kserve-6b59859755-rz7lr"]
Sep 15 10:39:55.785697 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:39:55.785659    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" path="/var/lib/kubelet/pods/18fe73b5-d76f-490a-bcb6-a9bdb8a942d9/volumes"
Sep 15 10:40:05.465859 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.465824    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"]
Sep 15 10:40:05.466480 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.466314    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" containerName="main"
Sep 15 10:40:05.466480 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.466333    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" containerName="main"
Sep 15 10:40:05.466480 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.466451    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fe73b5-d76f-490a-bcb6-a9bdb8a942d9" containerName="main"
Sep 15 10:40:05.470787 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.470764    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:05.472958 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.472934    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-44134e-serving-cert\""
Sep 15 10:40:05.480160 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.480131    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"]
Sep 15 10:40:05.544086 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.544042    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:05.544264 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.544094    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms2hq\" (UniqueName: \"kubernetes.io/projected/56d52716-58b7-4e1e-b1e6-6fe03046e008-kube-api-access-ms2hq\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:05.544264 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.544150    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d52716-58b7-4e1e-b1e6-6fe03046e008-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:05.544264 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.544229    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:05.645491 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.645451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:05.645686 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:40:05.645610    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-serving-cert: secret "maas-api-e2e-shared-b-w3-44134e-serving-cert" not found
Sep 15 10:40:05.645686 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.645631    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:05.645686 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:40:05.645675    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-maas-api-tls podName:56d52716-58b7-4e1e-b1e6-6fe03046e008 nodeName:}" failed. No retries permitted until 2026-09-15 10:40:06.145658273 +0000 UTC m=+1324.839813931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-maas-api-tls") pod "maas-api-e2e-shared-b-w3-44134e-574787566c-xv644" (UID: "56d52716-58b7-4e1e-b1e6-6fe03046e008") : secret "maas-api-e2e-shared-b-w3-44134e-serving-cert" not found
Sep 15 10:40:05.645858 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.645699    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms2hq\" (UniqueName: \"kubernetes.io/projected/56d52716-58b7-4e1e-b1e6-6fe03046e008-kube-api-access-ms2hq\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:05.645858 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.645728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d52716-58b7-4e1e-b1e6-6fe03046e008-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:05.646443 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.646427    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d52716-58b7-4e1e-b1e6-6fe03046e008-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:05.648097 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.648079    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:05.653855 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:05.653834    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms2hq\" (UniqueName: \"kubernetes.io/projected/56d52716-58b7-4e1e-b1e6-6fe03046e008-kube-api-access-ms2hq\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:06.068956 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.068921    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:40:06.092440 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.092400    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"]
Sep 15 10:40:06.096856 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.096831    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:40:06.099151 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.099127    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-44134e\""
Sep 15 10:40:06.099268 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.099196    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-44134e-dockercfg-9flcj\""
Sep 15 10:40:06.107113 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.107086    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"]
Sep 15 10:40:06.150513 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.150477    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:06.152955 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.152930    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-44134e-574787566c-xv644\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:06.251722 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.251682    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz92h\" (UniqueName: \"kubernetes.io/projected/2ec04348-d8fb-449a-ba54-67d04449a134-kube-api-access-nz92h\") pod \"payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht\" (UID: \"2ec04348-d8fb-449a-ba54-67d04449a134\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:40:06.251904 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.251759    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec04348-d8fb-449a-ba54-67d04449a134-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht\" (UID: \"2ec04348-d8fb-449a-ba54-67d04449a134\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:40:06.352766 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.352663    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz92h\" (UniqueName: \"kubernetes.io/projected/2ec04348-d8fb-449a-ba54-67d04449a134-kube-api-access-nz92h\") pod \"payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht\" (UID: \"2ec04348-d8fb-449a-ba54-67d04449a134\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:40:06.352766 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.352733    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec04348-d8fb-449a-ba54-67d04449a134-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht\" (UID: \"2ec04348-d8fb-449a-ba54-67d04449a134\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:40:06.353741 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.353714    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec04348-d8fb-449a-ba54-67d04449a134-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht\" (UID: \"2ec04348-d8fb-449a-ba54-67d04449a134\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:40:06.361144 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.361102    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz92h\" (UniqueName: \"kubernetes.io/projected/2ec04348-d8fb-449a-ba54-67d04449a134-kube-api-access-nz92h\") pod \"payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht\" (UID: \"2ec04348-d8fb-449a-ba54-67d04449a134\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:40:06.383197 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.383164    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:06.409487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.409457    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:40:06.669874 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.669767    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"]
Sep 15 10:40:06.681887 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:40:06.681849    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d52716_58b7_4e1e_b1e6_6fe03046e008.slice/crio-10a088dd3828cd0b3c20ae186a670d4c08ccfc0113773a36753dada41436931d WatchSource:0}: Error finding container 10a088dd3828cd0b3c20ae186a670d4c08ccfc0113773a36753dada41436931d: Status 404 returned error can't find the container with id 10a088dd3828cd0b3c20ae186a670d4c08ccfc0113773a36753dada41436931d
Sep 15 10:40:06.715610 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.715556    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"]
Sep 15 10:40:06.928718 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.928616    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644" event={"ID":"56d52716-58b7-4e1e-b1e6-6fe03046e008","Type":"ContainerStarted","Data":"10a088dd3828cd0b3c20ae186a670d4c08ccfc0113773a36753dada41436931d"}
Sep 15 10:40:06.930455 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.930417    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht" event={"ID":"2ec04348-d8fb-449a-ba54-67d04449a134","Type":"ContainerStarted","Data":"b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd"}
Sep 15 10:40:06.930602 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.930472    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht" event={"ID":"2ec04348-d8fb-449a-ba54-67d04449a134","Type":"ContainerStarted","Data":"fa90a7df0fe2b3d9ec21bbc700a2fd2a05b053b5fad8caf7b2d787989e46e69d"}
Sep 15 10:40:06.930602 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.930512    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:40:06.947199 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:06.947034    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht" podStartSLOduration=0.947011694 podStartE2EDuration="947.011694ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:40:06.946031589 +0000 UTC m=+1325.640187270" watchObservedRunningTime="2026-09-15 10:40:06.947011694 +0000 UTC m=+1325.641167409"
Sep 15 10:40:07.936615 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:07.936572    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644" event={"ID":"56d52716-58b7-4e1e-b1e6-6fe03046e008","Type":"ContainerStarted","Data":"b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343"}
Sep 15 10:40:07.937088 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:07.936748    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:07.952853 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:07.952773    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644" podStartSLOduration=2.493974046 podStartE2EDuration="2.952757156s" totalImagesPullingTime="458.78311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:40:07.952366829 +0000 UTC m=+1326.646522512" watchObservedRunningTime="2026-09-15 10:40:07.952757156 +0000 UTC m=+1326.646912857"
Sep 15 10:40:13.946995 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:13.946962    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:40:15.247927 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.247654    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"]
Sep 15 10:40:15.248424 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.247945    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk" podUID="85cf880f-e9a2-438f-a867-8a1e8e93b25a" containerName="payload-pre-processing" containerID="cri-o://7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727" gracePeriod=30
Sep 15 10:40:15.499814 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.499739    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:40:15.551121 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.551089    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rglh9\" (UniqueName: \"kubernetes.io/projected/85cf880f-e9a2-438f-a867-8a1e8e93b25a-kube-api-access-rglh9\") pod \"85cf880f-e9a2-438f-a867-8a1e8e93b25a\" (UID: \"85cf880f-e9a2-438f-a867-8a1e8e93b25a\") "
Sep 15 10:40:15.551312 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.551139    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cf880f-e9a2-438f-a867-8a1e8e93b25a-plugins-config-volume\") pod \"85cf880f-e9a2-438f-a867-8a1e8e93b25a\" (UID: \"85cf880f-e9a2-438f-a867-8a1e8e93b25a\") "
Sep 15 10:40:15.551594 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.551566    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85cf880f-e9a2-438f-a867-8a1e8e93b25a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85cf880f-e9a2-438f-a867-8a1e8e93b25a" (UID: "85cf880f-e9a2-438f-a867-8a1e8e93b25a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:40:15.553223 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.553198    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cf880f-e9a2-438f-a867-8a1e8e93b25a-kube-api-access-rglh9" (OuterVolumeSpecName: "kube-api-access-rglh9") pod "85cf880f-e9a2-438f-a867-8a1e8e93b25a" (UID: "85cf880f-e9a2-438f-a867-8a1e8e93b25a"). InnerVolumeSpecName "kube-api-access-rglh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:40:15.652966 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.652923    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rglh9\" (UniqueName: \"kubernetes.io/projected/85cf880f-e9a2-438f-a867-8a1e8e93b25a-kube-api-access-rglh9\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:40:15.652966 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.652958    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cf880f-e9a2-438f-a867-8a1e8e93b25a-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:40:15.975639 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.975601    2584 generic.go:356] "Generic (PLEG): container finished" podID="85cf880f-e9a2-438f-a867-8a1e8e93b25a" containerID="7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727" exitCode=0
Sep 15 10:40:15.975845 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.975649    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"
Sep 15 10:40:15.975845 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.975681    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk" event={"ID":"85cf880f-e9a2-438f-a867-8a1e8e93b25a","Type":"ContainerDied","Data":"7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727"}
Sep 15 10:40:15.975845 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.975732    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk" event={"ID":"85cf880f-e9a2-438f-a867-8a1e8e93b25a","Type":"ContainerDied","Data":"92f7fc3a75b83a2cca44798e1fb18a19682429504bb4d9b03f1fb623e25d9265"}
Sep 15 10:40:15.975845 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.975761    2584 scope.go:117] "RemoveContainer" containerID="7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727"
Sep 15 10:40:15.984736 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.984715    2584 scope.go:117] "RemoveContainer" containerID="7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727"
Sep 15 10:40:15.985048 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:40:15.985026    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727\": container with ID starting with 7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727 not found: ID does not exist" containerID="7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727"
Sep 15 10:40:15.985103 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.985058    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727"} err="failed to get container status \"7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727\": rpc error: code = NotFound desc = could not find container \"7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727\": container with ID starting with 7ed71d867cc48328bf335fbf4d78f0c0750e8df0079c9271b84bf081add07727 not found: ID does not exist"
Sep 15 10:40:15.991856 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.991829    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"]
Sep 15 10:40:15.995651 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:15.995625    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6111ecc0-96f4ccc7d-qjpjk"]
Sep 15 10:40:17.784993 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:17.784962    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85cf880f-e9a2-438f-a867-8a1e8e93b25a" path="/var/lib/kubelet/pods/85cf880f-e9a2-438f-a867-8a1e8e93b25a/volumes"
Sep 15 10:40:17.939275 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:17.939249    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:40:35.236269 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:35.236185    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"]
Sep 15 10:40:45.463731 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:45.463692    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:40:49.253927 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:49.253893    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"]
Sep 15 10:40:50.255861 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.255774    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl" podUID="e696aa28-7767-43f8-b8d9-61765f8436f6" containerName="main" containerID="cri-o://0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22" gracePeriod=45
Sep 15 10:40:50.501172 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.501141    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:40:50.599112 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599028    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-dshm\") pod \"e696aa28-7767-43f8-b8d9-61765f8436f6\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") "
Sep 15 10:40:50.599112 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599066    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-home\") pod \"e696aa28-7767-43f8-b8d9-61765f8436f6\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") "
Sep 15 10:40:50.599325 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599134    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-model-cache\") pod \"e696aa28-7767-43f8-b8d9-61765f8436f6\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") "
Sep 15 10:40:50.599325 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599177    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-tmp-dir\") pod \"e696aa28-7767-43f8-b8d9-61765f8436f6\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") "
Sep 15 10:40:50.599325 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599214    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp2kn\" (UniqueName: \"kubernetes.io/projected/e696aa28-7767-43f8-b8d9-61765f8436f6-kube-api-access-kp2kn\") pod \"e696aa28-7767-43f8-b8d9-61765f8436f6\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") "
Sep 15 10:40:50.599325 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599274    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e696aa28-7767-43f8-b8d9-61765f8436f6-tls-certs\") pod \"e696aa28-7767-43f8-b8d9-61765f8436f6\" (UID: \"e696aa28-7767-43f8-b8d9-61765f8436f6\") "
Sep 15 10:40:50.599542 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599379    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-home" (OuterVolumeSpecName: "home") pod "e696aa28-7767-43f8-b8d9-61765f8436f6" (UID: "e696aa28-7767-43f8-b8d9-61765f8436f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:40:50.599542 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599411    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e696aa28-7767-43f8-b8d9-61765f8436f6" (UID: "e696aa28-7767-43f8-b8d9-61765f8436f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:40:50.599542 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599435    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e696aa28-7767-43f8-b8d9-61765f8436f6" (UID: "e696aa28-7767-43f8-b8d9-61765f8436f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:40:50.599721 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599698    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:40:50.599825 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599722    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:40:50.599825 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.599734    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:40:50.601551 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.601524    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-dshm" (OuterVolumeSpecName: "dshm") pod "e696aa28-7767-43f8-b8d9-61765f8436f6" (UID: "e696aa28-7767-43f8-b8d9-61765f8436f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:40:50.601682 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.601619    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e696aa28-7767-43f8-b8d9-61765f8436f6-kube-api-access-kp2kn" (OuterVolumeSpecName: "kube-api-access-kp2kn") pod "e696aa28-7767-43f8-b8d9-61765f8436f6" (UID: "e696aa28-7767-43f8-b8d9-61765f8436f6"). InnerVolumeSpecName "kube-api-access-kp2kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:40:50.601682 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.601633    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e696aa28-7767-43f8-b8d9-61765f8436f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e696aa28-7767-43f8-b8d9-61765f8436f6" (UID: "e696aa28-7767-43f8-b8d9-61765f8436f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:40:50.700556 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.700524    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e696aa28-7767-43f8-b8d9-61765f8436f6-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:40:50.700556 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.700550    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2kn\" (UniqueName: \"kubernetes.io/projected/e696aa28-7767-43f8-b8d9-61765f8436f6-kube-api-access-kp2kn\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:40:50.700556 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:50.700561    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e696aa28-7767-43f8-b8d9-61765f8436f6-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:40:51.154513 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:51.154473    2584 generic.go:356] "Generic (PLEG): container finished" podID="e696aa28-7767-43f8-b8d9-61765f8436f6" containerID="0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22" exitCode=0
Sep 15 10:40:51.154704 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:51.154527    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"
Sep 15 10:40:51.154704 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:51.154565    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl" event={"ID":"e696aa28-7767-43f8-b8d9-61765f8436f6","Type":"ContainerDied","Data":"0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22"}
Sep 15 10:40:51.154704 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:51.154625    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl" event={"ID":"e696aa28-7767-43f8-b8d9-61765f8436f6","Type":"ContainerDied","Data":"023e41d3577eea6f0ff379444a949a374047685c06efb11ba5098ec1565e7688"}
Sep 15 10:40:51.154704 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:51.154646    2584 scope.go:117] "RemoveContainer" containerID="0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22"
Sep 15 10:40:51.163353 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:51.163337    2584 scope.go:117] "RemoveContainer" containerID="0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22"
Sep 15 10:40:51.163562 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:40:51.163547    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22\": container with ID starting with 0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22 not found: ID does not exist" containerID="0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22"
Sep 15 10:40:51.163610 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:51.163569    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22"} err="failed to get container status \"0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22\": rpc error: code = NotFound desc = could not find container \"0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22\": container with ID starting with 0b31d7f8ca9640e822cacaec0254d95ed6e0e30bc4550560abdbb34e16e75b22 not found: ID does not exist"
Sep 15 10:40:51.177331 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:51.177306    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"]
Sep 15 10:40:51.181152 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:51.181126    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-b-55001805-kserve-6c5bc8868c-fnvpl"]
Sep 15 10:40:51.786178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:40:51.786146    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e696aa28-7767-43f8-b8d9-61765f8436f6" path="/var/lib/kubelet/pods/e696aa28-7767-43f8-b8d9-61765f8436f6/volumes"
Sep 15 10:41:04.274853 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.274785    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg" podUID="d067108e-9ede-4f74-b36f-6c0237057c1b" containerName="main" containerID="cri-o://6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d" gracePeriod=45
Sep 15 10:41:04.527251 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.527184    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:41:04.633711 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.633673    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-model-cache\") pod \"d067108e-9ede-4f74-b36f-6c0237057c1b\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") "
Sep 15 10:41:04.633922 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.633739    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf999\" (UniqueName: \"kubernetes.io/projected/d067108e-9ede-4f74-b36f-6c0237057c1b-kube-api-access-gf999\") pod \"d067108e-9ede-4f74-b36f-6c0237057c1b\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") "
Sep 15 10:41:04.633922 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.633819    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-home\") pod \"d067108e-9ede-4f74-b36f-6c0237057c1b\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") "
Sep 15 10:41:04.633922 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.633880    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-tmp-dir\") pod \"d067108e-9ede-4f74-b36f-6c0237057c1b\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") "
Sep 15 10:41:04.633922 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.633911    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d067108e-9ede-4f74-b36f-6c0237057c1b-tls-certs\") pod \"d067108e-9ede-4f74-b36f-6c0237057c1b\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") "
Sep 15 10:41:04.634157 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.633944    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-dshm\") pod \"d067108e-9ede-4f74-b36f-6c0237057c1b\" (UID: \"d067108e-9ede-4f74-b36f-6c0237057c1b\") "
Sep 15 10:41:04.634157 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.634002    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d067108e-9ede-4f74-b36f-6c0237057c1b" (UID: "d067108e-9ede-4f74-b36f-6c0237057c1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:41:04.634157 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.634127    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-home" (OuterVolumeSpecName: "home") pod "d067108e-9ede-4f74-b36f-6c0237057c1b" (UID: "d067108e-9ede-4f74-b36f-6c0237057c1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:41:04.634157 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.634140    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d067108e-9ede-4f74-b36f-6c0237057c1b" (UID: "d067108e-9ede-4f74-b36f-6c0237057c1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:41:04.634292 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.634271    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:41:04.634292 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.634285    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:41:04.634400 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.634293    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:41:04.636064 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.636041    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d067108e-9ede-4f74-b36f-6c0237057c1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d067108e-9ede-4f74-b36f-6c0237057c1b" (UID: "d067108e-9ede-4f74-b36f-6c0237057c1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:41:04.636396 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.636368    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-dshm" (OuterVolumeSpecName: "dshm") pod "d067108e-9ede-4f74-b36f-6c0237057c1b" (UID: "d067108e-9ede-4f74-b36f-6c0237057c1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:41:04.636396 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.636380    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d067108e-9ede-4f74-b36f-6c0237057c1b-kube-api-access-gf999" (OuterVolumeSpecName: "kube-api-access-gf999") pod "d067108e-9ede-4f74-b36f-6c0237057c1b" (UID: "d067108e-9ede-4f74-b36f-6c0237057c1b"). InnerVolumeSpecName "kube-api-access-gf999". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:41:04.735135 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.735082    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf999\" (UniqueName: \"kubernetes.io/projected/d067108e-9ede-4f74-b36f-6c0237057c1b-kube-api-access-gf999\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:41:04.735135 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.735128    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d067108e-9ede-4f74-b36f-6c0237057c1b-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:41:04.735135 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:04.735139    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d067108e-9ede-4f74-b36f-6c0237057c1b-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:41:05.227505 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:05.227469    2584 generic.go:356] "Generic (PLEG): container finished" podID="d067108e-9ede-4f74-b36f-6c0237057c1b" containerID="6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d" exitCode=0
Sep 15 10:41:05.227692 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:05.227512    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"
Sep 15 10:41:05.227692 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:05.227555    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg" event={"ID":"d067108e-9ede-4f74-b36f-6c0237057c1b","Type":"ContainerDied","Data":"6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d"}
Sep 15 10:41:05.227692 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:05.227600    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg" event={"ID":"d067108e-9ede-4f74-b36f-6c0237057c1b","Type":"ContainerDied","Data":"f4f4c96ab9a88c1f3391ec67eccce60582dfd89f9566148e8028790771770fbb"}
Sep 15 10:41:05.227692 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:05.227621    2584 scope.go:117] "RemoveContainer" containerID="6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d"
Sep 15 10:41:05.236715 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:05.236695    2584 scope.go:117] "RemoveContainer" containerID="6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d"
Sep 15 10:41:05.237000 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:41:05.236977    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d\": container with ID starting with 6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d not found: ID does not exist" containerID="6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d"
Sep 15 10:41:05.237086 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:05.237009    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d"} err="failed to get container status \"6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d\": rpc error: code = NotFound desc = could not find container \"6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d\": container with ID starting with 6aee515c78efdf37561958b80418c6ae88ae389b01a1aa9935702bf434645d3d not found: ID does not exist"
Sep 15 10:41:05.247770 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:05.247739    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"]
Sep 15 10:41:05.251072 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:05.251051    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/e2e-conflict-a-55001805-kserve-6489664f9c-rnsrg"]
Sep 15 10:41:05.785167 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:05.785134    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d067108e-9ede-4f74-b36f-6c0237057c1b" path="/var/lib/kubelet/pods/d067108e-9ede-4f74-b36f-6c0237057c1b/volumes"
Sep 15 10:41:16.255119 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:16.255071    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:41:41.444955 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.444913    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"]
Sep 15 10:41:41.445537 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.445515    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85cf880f-e9a2-438f-a867-8a1e8e93b25a" containerName="payload-pre-processing"
Sep 15 10:41:41.445582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.445543    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="85cf880f-e9a2-438f-a867-8a1e8e93b25a" containerName="payload-pre-processing"
Sep 15 10:41:41.445582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.445567    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e696aa28-7767-43f8-b8d9-61765f8436f6" containerName="main"
Sep 15 10:41:41.445582 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.445576    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e696aa28-7767-43f8-b8d9-61765f8436f6" containerName="main"
Sep 15 10:41:41.445708 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.445594    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d067108e-9ede-4f74-b36f-6c0237057c1b" containerName="main"
Sep 15 10:41:41.445708 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.445603    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d067108e-9ede-4f74-b36f-6c0237057c1b" containerName="main"
Sep 15 10:41:41.445708 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.445690    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e696aa28-7767-43f8-b8d9-61765f8436f6" containerName="main"
Sep 15 10:41:41.445879 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.445710    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="85cf880f-e9a2-438f-a867-8a1e8e93b25a" containerName="payload-pre-processing"
Sep 15 10:41:41.445879 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.445719    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d067108e-9ede-4f74-b36f-6c0237057c1b" containerName="main"
Sep 15 10:41:41.449233 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.449205    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.452111 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.451977    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ff0655\"/\"distinct-w4-ff0655-kserve-self-signed-certs\""
Sep 15 10:41:41.455260 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.455234    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"]
Sep 15 10:41:41.490538 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.490508    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-dshm\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.490684 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.490544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tls-certs\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.490684 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.490566    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-model-cache\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.490684 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.490650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-home\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.490684 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.490683    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28lsg\" (UniqueName: \"kubernetes.io/projected/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-kube-api-access-28lsg\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.490909 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.490703    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tmp-dir\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.592197 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.592168    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tmp-dir\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.592381 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.592241    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-dshm\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.592381 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.592335    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tls-certs\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.592381 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.592357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-model-cache\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.592559 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.592410    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-home\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.592559 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.592455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lsg\" (UniqueName: \"kubernetes.io/projected/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-kube-api-access-28lsg\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.592654 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.592554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tmp-dir\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.592692 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.592658    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-model-cache\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.592792 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.592768    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-home\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.594450 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.594423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-dshm\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.594758 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.594741    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tls-certs\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.600522 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.600497    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lsg\" (UniqueName: \"kubernetes.io/projected/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-kube-api-access-28lsg\") pod \"distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:41.761129 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:41.761096    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:42.031100 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:42.031071    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"]
Sep 15 10:41:42.033246 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:41:42.033220    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cc17d65_a9c9_4e35_b2d8_41dddfd31759.slice/crio-90b96026c925a85d026e929c03b07cf3e2d0a0cac68bf514ca7b87f3969a8279 WatchSource:0}: Error finding container 90b96026c925a85d026e929c03b07cf3e2d0a0cac68bf514ca7b87f3969a8279: Status 404 returned error can't find the container with id 90b96026c925a85d026e929c03b07cf3e2d0a0cac68bf514ca7b87f3969a8279
Sep 15 10:41:42.410983 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:42.410888    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj" event={"ID":"7cc17d65-a9c9-4e35-b2d8-41dddfd31759","Type":"ContainerStarted","Data":"ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21"}
Sep 15 10:41:42.410983 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:42.410932    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj" event={"ID":"7cc17d65-a9c9-4e35-b2d8-41dddfd31759","Type":"ContainerStarted","Data":"90b96026c925a85d026e929c03b07cf3e2d0a0cac68bf514ca7b87f3969a8279"}
Sep 15 10:41:42.427118 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:42.427069    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj" podStartSLOduration=1.42705671 podStartE2EDuration="1.42705671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:41:42.426175967 +0000 UTC m=+1421.120331647" watchObservedRunningTime="2026-09-15 10:41:42.42705671 +0000 UTC m=+1421.121212389"
Sep 15 10:41:45.150737 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:45.150701    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:41:51.761924 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:51.761891    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:51.762454 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:51.762039    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:51.774932 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:51.774912    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:41:52.471741 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:41:52.471712    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:42:01.922366 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:01.922289    2584 scope.go:117] "RemoveContainer" containerID="0c738df31376516a867ab9ce985729829395d05b7e323d0efd3e634936862354"
Sep 15 10:42:21.467741 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:21.467696    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:42:44.652222 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.652189    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"]
Sep 15 10:42:44.655710 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.655692    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.657959 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.657937    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8c1047\"/\"embedding-w0-8c1047-kserve-self-signed-certs\""
Sep 15 10:42:44.663896 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.663872    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"]
Sep 15 10:42:44.826378 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.826340    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7r86\" (UniqueName: \"kubernetes.io/projected/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-kube-api-access-r7r86\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.826590 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.826409    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tls-certs\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.826590 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.826485    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-model-cache\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.826590 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.826524    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-dshm\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.826590 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.826567    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-home\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.826761 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.826619    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tmp-dir\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.928056 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.927973    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tls-certs\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.928056 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.928023    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-model-cache\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.928241 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.928135    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-dshm\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.928241 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.928171    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-home\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.928241 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.928225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tmp-dir\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.928398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.928277    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7r86\" (UniqueName: \"kubernetes.io/projected/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-kube-api-access-r7r86\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.928462 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.928395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-model-cache\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.928585 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.928563    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-home\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.928669 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.928646    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tmp-dir\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.930423 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.930398    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-dshm\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.930556 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.930542    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tls-certs\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.935263 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.935240    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7r86\" (UniqueName: \"kubernetes.io/projected/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-kube-api-access-r7r86\") pod \"embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") " pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:44.965418 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:44.965396    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:45.252164 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:45.252135    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"]
Sep 15 10:42:45.255795 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:42:45.255760    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e6a8377_e9ba_4d65_9e4e_e55541d6c288.slice/crio-09790ebccebf989a0faec7e6885450bd964117bfcdae76c6a0dfce5c94f80974 WatchSource:0}: Error finding container 09790ebccebf989a0faec7e6885450bd964117bfcdae76c6a0dfce5c94f80974: Status 404 returned error can't find the container with id 09790ebccebf989a0faec7e6885450bd964117bfcdae76c6a0dfce5c94f80974
Sep 15 10:42:45.719727 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:45.719610    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt" event={"ID":"2e6a8377-e9ba-4d65-9e4e-e55541d6c288","Type":"ContainerStarted","Data":"86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752"}
Sep 15 10:42:45.719727 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:45.719680    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt" event={"ID":"2e6a8377-e9ba-4d65-9e4e-e55541d6c288","Type":"ContainerStarted","Data":"09790ebccebf989a0faec7e6885450bd964117bfcdae76c6a0dfce5c94f80974"}
Sep 15 10:42:45.738926 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:45.738877    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt" podStartSLOduration=1.738863808 podStartE2EDuration="1.738863808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:42:45.737978963 +0000 UTC m=+1484.432134645" watchObservedRunningTime="2026-09-15 10:42:45.738863808 +0000 UTC m=+1484.433019488"
Sep 15 10:42:54.966065 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:54.966011    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:54.966580 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:54.966083    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:54.978746 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:54.978717    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:55.785376 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:55.785349    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:42:56.257717 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:42:56.257684    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:43:01.234490 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.234450    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"]
Sep 15 10:43:01.238118 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.238100    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.240254 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.240233    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ff0655\"/\"distinct-2-w4-ff0655-kserve-self-signed-certs\""
Sep 15 10:43:01.244387 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.244363    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"]
Sep 15 10:43:01.291093 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.291037    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkzhn\" (UniqueName: \"kubernetes.io/projected/9284f93f-ee7f-4d43-879d-a169df42db13-kube-api-access-dkzhn\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.291286 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.291151    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-model-cache\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.291286 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.291224    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-tmp-dir\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.291286 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.291271    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9284f93f-ee7f-4d43-879d-a169df42db13-tls-certs\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.291416 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.291303    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-home\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.291416 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.291338    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-dshm\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.392411 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.392370    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-model-cache\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.392598 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.392421    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-tmp-dir\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.392598 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.392444    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9284f93f-ee7f-4d43-879d-a169df42db13-tls-certs\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.392598 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.392471    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-home\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.392598 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.392506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-dshm\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.392851 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.392673    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzhn\" (UniqueName: \"kubernetes.io/projected/9284f93f-ee7f-4d43-879d-a169df42db13-kube-api-access-dkzhn\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.392919 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.392856    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-model-cache\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.392975 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.392923    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-home\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.393033 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.393016    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-tmp-dir\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.394858 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.394832    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-dshm\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.394978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.394949    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9284f93f-ee7f-4d43-879d-a169df42db13-tls-certs\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.401642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.401619    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkzhn\" (UniqueName: \"kubernetes.io/projected/9284f93f-ee7f-4d43-879d-a169df42db13-kube-api-access-dkzhn\") pod \"distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") " pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.549500 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.549459    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:01.823519 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:01.823439    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"]
Sep 15 10:43:01.826625 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:43:01.826601    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9284f93f_ee7f_4d43_879d_a169df42db13.slice/crio-eeff2c19c2ac5f11911ad48c4fd4867e45ce119f6defab3e94b17db4b525f7a0 WatchSource:0}: Error finding container eeff2c19c2ac5f11911ad48c4fd4867e45ce119f6defab3e94b17db4b525f7a0: Status 404 returned error can't find the container with id eeff2c19c2ac5f11911ad48c4fd4867e45ce119f6defab3e94b17db4b525f7a0
Sep 15 10:43:02.812006 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:02.811965    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n" event={"ID":"9284f93f-ee7f-4d43-879d-a169df42db13","Type":"ContainerStarted","Data":"bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8"}
Sep 15 10:43:02.812401 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:02.812017    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n" event={"ID":"9284f93f-ee7f-4d43-879d-a169df42db13","Type":"ContainerStarted","Data":"eeff2c19c2ac5f11911ad48c4fd4867e45ce119f6defab3e94b17db4b525f7a0"}
Sep 15 10:43:02.826678 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:02.826635    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n" podStartSLOduration=1.826621568 podStartE2EDuration="1.826621568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:43:02.825926441 +0000 UTC m=+1501.520082137" watchObservedRunningTime="2026-09-15 10:43:02.826621568 +0000 UTC m=+1501.520777247"
Sep 15 10:43:11.549920 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:11.549882    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:11.550552 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:11.549938    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:11.562617 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:11.562590    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:11.869109 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:11.869020    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:43:23.397923 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.397879    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"]
Sep 15 10:43:23.401599 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.401581    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:43:23.404017 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.403995    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-069380\""
Sep 15 10:43:23.411432 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.411403    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"]
Sep 15 10:43:23.441764 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.441724    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"]
Sep 15 10:43:23.445164 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.445144    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:43:23.447452 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.447427    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-069380-dockercfg-nlbf4\""
Sep 15 10:43:23.453460 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.453436    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"]
Sep 15 10:43:23.511031 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.510993    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kc4n\" (UniqueName: \"kubernetes.io/projected/62739fd7-9067-46b6-83ee-7b5e398f839a-kube-api-access-8kc4n\") pod \"payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl\" (UID: \"62739fd7-9067-46b6-83ee-7b5e398f839a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:43:23.511202 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.511089    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62739fd7-9067-46b6-83ee-7b5e398f839a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl\" (UID: \"62739fd7-9067-46b6-83ee-7b5e398f839a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:43:23.612660 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.612621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e01aeaff-8342-4127-b7db-ed523673e74c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-069380-75dc6574df-cvxvq\" (UID: \"e01aeaff-8342-4127-b7db-ed523673e74c\") " pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:43:23.612660 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.612667    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62739fd7-9067-46b6-83ee-7b5e398f839a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl\" (UID: \"62739fd7-9067-46b6-83ee-7b5e398f839a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:43:23.612895 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.612865    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc4n\" (UniqueName: \"kubernetes.io/projected/62739fd7-9067-46b6-83ee-7b5e398f839a-kube-api-access-8kc4n\") pod \"payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl\" (UID: \"62739fd7-9067-46b6-83ee-7b5e398f839a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:43:23.612942 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.612923    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzwv6\" (UniqueName: \"kubernetes.io/projected/e01aeaff-8342-4127-b7db-ed523673e74c-kube-api-access-xzwv6\") pod \"payload-processing-e2e-api-b-069380-75dc6574df-cvxvq\" (UID: \"e01aeaff-8342-4127-b7db-ed523673e74c\") " pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:43:23.613468 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.613447    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62739fd7-9067-46b6-83ee-7b5e398f839a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl\" (UID: \"62739fd7-9067-46b6-83ee-7b5e398f839a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:43:23.620655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.620634    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kc4n\" (UniqueName: \"kubernetes.io/projected/62739fd7-9067-46b6-83ee-7b5e398f839a-kube-api-access-8kc4n\") pod \"payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl\" (UID: \"62739fd7-9067-46b6-83ee-7b5e398f839a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:43:23.714121 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.714016    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwv6\" (UniqueName: \"kubernetes.io/projected/e01aeaff-8342-4127-b7db-ed523673e74c-kube-api-access-xzwv6\") pod \"payload-processing-e2e-api-b-069380-75dc6574df-cvxvq\" (UID: \"e01aeaff-8342-4127-b7db-ed523673e74c\") " pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:43:23.714121 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.714100    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e01aeaff-8342-4127-b7db-ed523673e74c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-069380-75dc6574df-cvxvq\" (UID: \"e01aeaff-8342-4127-b7db-ed523673e74c\") " pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:43:23.714121 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.714111    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:43:23.714822 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.714781    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e01aeaff-8342-4127-b7db-ed523673e74c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-069380-75dc6574df-cvxvq\" (UID: \"e01aeaff-8342-4127-b7db-ed523673e74c\") " pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:43:23.721410 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.721384    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwv6\" (UniqueName: \"kubernetes.io/projected/e01aeaff-8342-4127-b7db-ed523673e74c-kube-api-access-xzwv6\") pod \"payload-processing-e2e-api-b-069380-75dc6574df-cvxvq\" (UID: \"e01aeaff-8342-4127-b7db-ed523673e74c\") " pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:43:23.757300 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.757265    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:43:23.984082 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:23.984054    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"]
Sep 15 10:43:23.986017 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:43:23.985989    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62739fd7_9067_46b6_83ee_7b5e398f839a.slice/crio-df5ee49d8330ed4db852639254a43c18b35584b87d0bce7fbd863c9f7b5f2856 WatchSource:0}: Error finding container df5ee49d8330ed4db852639254a43c18b35584b87d0bce7fbd863c9f7b5f2856: Status 404 returned error can't find the container with id df5ee49d8330ed4db852639254a43c18b35584b87d0bce7fbd863c9f7b5f2856
Sep 15 10:43:24.043473 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:24.043448    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"]
Sep 15 10:43:24.045323 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:43:24.045298    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01aeaff_8342_4127_b7db_ed523673e74c.slice/crio-71b52d0fc13411c1ed31d0bd8299d118d999d09bc7c79a94534006ec57f95750 WatchSource:0}: Error finding container 71b52d0fc13411c1ed31d0bd8299d118d999d09bc7c79a94534006ec57f95750: Status 404 returned error can't find the container with id 71b52d0fc13411c1ed31d0bd8299d118d999d09bc7c79a94534006ec57f95750
Sep 15 10:43:24.937863 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:24.937822    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl" event={"ID":"62739fd7-9067-46b6-83ee-7b5e398f839a","Type":"ContainerStarted","Data":"3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3"}
Sep 15 10:43:24.938335 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:24.937890    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl" event={"ID":"62739fd7-9067-46b6-83ee-7b5e398f839a","Type":"ContainerStarted","Data":"df5ee49d8330ed4db852639254a43c18b35584b87d0bce7fbd863c9f7b5f2856"}
Sep 15 10:43:24.938335 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:24.937918    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:43:24.939739 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:24.939709    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq" event={"ID":"e01aeaff-8342-4127-b7db-ed523673e74c","Type":"ContainerStarted","Data":"2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2"}
Sep 15 10:43:24.939919 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:24.939763    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq" event={"ID":"e01aeaff-8342-4127-b7db-ed523673e74c","Type":"ContainerStarted","Data":"71b52d0fc13411c1ed31d0bd8299d118d999d09bc7c79a94534006ec57f95750"}
Sep 15 10:43:24.939919 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:24.939791    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:43:24.953653 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:24.953595    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl" podStartSLOduration=1.953580031 podStartE2EDuration="1.953580031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:43:24.952106985 +0000 UTC m=+1523.646262665" watchObservedRunningTime="2026-09-15 10:43:24.953580031 +0000 UTC m=+1523.647735710"
Sep 15 10:43:24.966740 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:24.966693    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq" podStartSLOduration=1.966678424 podStartE2EDuration="1.966678424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:43:24.965710986 +0000 UTC m=+1523.659866668" watchObservedRunningTime="2026-09-15 10:43:24.966678424 +0000 UTC m=+1523.660834149"
Sep 15 10:43:35.946791 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:35.946758    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:43:35.949265 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:35.946850    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:43:38.858374 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:43:38.858339    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:44:18.957200 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:18.957159    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:44:27.811946 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:27.811894    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"]
Sep 15 10:44:27.812412 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:27.812153    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq" podUID="e01aeaff-8342-4127-b7db-ed523673e74c" containerName="payload-processing" containerID="cri-o://2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2" gracePeriod=30
Sep 15 10:44:27.819123 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:27.819094    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"]
Sep 15 10:44:27.819512 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:27.819483    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl" podUID="62739fd7-9067-46b6-83ee-7b5e398f839a" containerName="payload-pre-processing" containerID="cri-o://3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3" gracePeriod=30
Sep 15 10:44:28.084144 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.084123    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:44:28.098503 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.098480    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:44:28.166567 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.166535    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e01aeaff-8342-4127-b7db-ed523673e74c-plugins-config-volume\") pod \"e01aeaff-8342-4127-b7db-ed523673e74c\" (UID: \"e01aeaff-8342-4127-b7db-ed523673e74c\") "
Sep 15 10:44:28.166567 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.166574    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzwv6\" (UniqueName: \"kubernetes.io/projected/e01aeaff-8342-4127-b7db-ed523673e74c-kube-api-access-xzwv6\") pod \"e01aeaff-8342-4127-b7db-ed523673e74c\" (UID: \"e01aeaff-8342-4127-b7db-ed523673e74c\") "
Sep 15 10:44:28.166788 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.166637    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62739fd7-9067-46b6-83ee-7b5e398f839a-plugins-config-volume\") pod \"62739fd7-9067-46b6-83ee-7b5e398f839a\" (UID: \"62739fd7-9067-46b6-83ee-7b5e398f839a\") "
Sep 15 10:44:28.166788 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.166665    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kc4n\" (UniqueName: \"kubernetes.io/projected/62739fd7-9067-46b6-83ee-7b5e398f839a-kube-api-access-8kc4n\") pod \"62739fd7-9067-46b6-83ee-7b5e398f839a\" (UID: \"62739fd7-9067-46b6-83ee-7b5e398f839a\") "
Sep 15 10:44:28.167023 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.166998    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62739fd7-9067-46b6-83ee-7b5e398f839a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62739fd7-9067-46b6-83ee-7b5e398f839a" (UID: "62739fd7-9067-46b6-83ee-7b5e398f839a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:44:28.167086 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.167005    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01aeaff-8342-4127-b7db-ed523673e74c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e01aeaff-8342-4127-b7db-ed523673e74c" (UID: "e01aeaff-8342-4127-b7db-ed523673e74c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:44:28.168733 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.168711    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01aeaff-8342-4127-b7db-ed523673e74c-kube-api-access-xzwv6" (OuterVolumeSpecName: "kube-api-access-xzwv6") pod "e01aeaff-8342-4127-b7db-ed523673e74c" (UID: "e01aeaff-8342-4127-b7db-ed523673e74c"). InnerVolumeSpecName "kube-api-access-xzwv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:44:28.168831 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.168741    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62739fd7-9067-46b6-83ee-7b5e398f839a-kube-api-access-8kc4n" (OuterVolumeSpecName: "kube-api-access-8kc4n") pod "62739fd7-9067-46b6-83ee-7b5e398f839a" (UID: "62739fd7-9067-46b6-83ee-7b5e398f839a"). InnerVolumeSpecName "kube-api-access-8kc4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:44:28.256097 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.256062    2584 generic.go:356] "Generic (PLEG): container finished" podID="62739fd7-9067-46b6-83ee-7b5e398f839a" containerID="3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3" exitCode=0
Sep 15 10:44:28.256284 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.256107    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"
Sep 15 10:44:28.256284 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.256145    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl" event={"ID":"62739fd7-9067-46b6-83ee-7b5e398f839a","Type":"ContainerDied","Data":"3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3"}
Sep 15 10:44:28.256284 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.256196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl" event={"ID":"62739fd7-9067-46b6-83ee-7b5e398f839a","Type":"ContainerDied","Data":"df5ee49d8330ed4db852639254a43c18b35584b87d0bce7fbd863c9f7b5f2856"}
Sep 15 10:44:28.256284 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.256218    2584 scope.go:117] "RemoveContainer" containerID="3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3"
Sep 15 10:44:28.257598 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.257573    2584 generic.go:356] "Generic (PLEG): container finished" podID="e01aeaff-8342-4127-b7db-ed523673e74c" containerID="2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2" exitCode=0
Sep 15 10:44:28.257720 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.257609    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"
Sep 15 10:44:28.257720 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.257614    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq" event={"ID":"e01aeaff-8342-4127-b7db-ed523673e74c","Type":"ContainerDied","Data":"2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2"}
Sep 15 10:44:28.257720 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.257653    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq" event={"ID":"e01aeaff-8342-4127-b7db-ed523673e74c","Type":"ContainerDied","Data":"71b52d0fc13411c1ed31d0bd8299d118d999d09bc7c79a94534006ec57f95750"}
Sep 15 10:44:28.267968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.267938    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e01aeaff-8342-4127-b7db-ed523673e74c-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:28.268118 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.267974    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzwv6\" (UniqueName: \"kubernetes.io/projected/e01aeaff-8342-4127-b7db-ed523673e74c-kube-api-access-xzwv6\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:28.268118 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.267989    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62739fd7-9067-46b6-83ee-7b5e398f839a-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:28.268118 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.268001    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kc4n\" (UniqueName: \"kubernetes.io/projected/62739fd7-9067-46b6-83ee-7b5e398f839a-kube-api-access-8kc4n\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:28.272228 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.272196    2584 scope.go:117] "RemoveContainer" containerID="3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3"
Sep 15 10:44:28.272591 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:44:28.272562    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3\": container with ID starting with 3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3 not found: ID does not exist" containerID="3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3"
Sep 15 10:44:28.272701 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.272600    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3"} err="failed to get container status \"3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3\": rpc error: code = NotFound desc = could not find container \"3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3\": container with ID starting with 3ac6e6e649025b728dc7bff381e1778720b5515b55e6009f30669a26be12cba3 not found: ID does not exist"
Sep 15 10:44:28.272701 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.272633    2584 scope.go:117] "RemoveContainer" containerID="2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2"
Sep 15 10:44:28.280824 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.280708    2584 scope.go:117] "RemoveContainer" containerID="2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2"
Sep 15 10:44:28.281079 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:44:28.281057    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2\": container with ID starting with 2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2 not found: ID does not exist" containerID="2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2"
Sep 15 10:44:28.281172 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.281084    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2"} err="failed to get container status \"2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2\": rpc error: code = NotFound desc = could not find container \"2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2\": container with ID starting with 2dec8f16323584e52002895e467047cd410ab094b680beac255ad01eb65521e2 not found: ID does not exist"
Sep 15 10:44:28.283203 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.283182    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"]
Sep 15 10:44:28.287261 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.287237    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-069380-75dc6574df-cvxvq"]
Sep 15 10:44:28.297637 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.297613    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"]
Sep 15 10:44:28.301519 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:28.301498    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-069380-69959dd87d-w65cl"]
Sep 15 10:44:29.785266 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:29.785234    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62739fd7-9067-46b6-83ee-7b5e398f839a" path="/var/lib/kubelet/pods/62739fd7-9067-46b6-83ee-7b5e398f839a/volumes"
Sep 15 10:44:29.785644 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:29.785578    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e01aeaff-8342-4127-b7db-ed523673e74c" path="/var/lib/kubelet/pods/e01aeaff-8342-4127-b7db-ed523673e74c/volumes"
Sep 15 10:44:30.456374 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:30.456338    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"]
Sep 15 10:44:30.461497 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:30.461469    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"]
Sep 15 10:44:30.474264 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:30.474236    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"]
Sep 15 10:44:30.483533 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:30.483504    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"]
Sep 15 10:44:45.476817 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.476759    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn" podUID="d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" containerName="main" containerID="cri-o://6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224" gracePeriod=45
Sep 15 10:44:45.487253 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.487194    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt" podUID="2e6a8377-e9ba-4d65-9e4e-e55541d6c288" containerName="main" containerID="cri-o://86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752" gracePeriod=45
Sep 15 10:44:45.503195 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.503162    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl" podUID="29d873a2-0f82-45a3-9fcf-cf193bde0e8c" containerName="main" containerID="cri-o://2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002" gracePeriod=45
Sep 15 10:44:45.503195 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.503163    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c" podUID="699599fa-ce28-4c30-983d-778af87e42f4" containerName="main" containerID="cri-o://9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd" gracePeriod=45
Sep 15 10:44:45.805915 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.805878    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:44:45.839747 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.839721    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-dshm\") pod \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") "
Sep 15 10:44:45.839922 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.839759    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-model-cache\") pod \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") "
Sep 15 10:44:45.839922 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.839849    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tmp-dir\") pod \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") "
Sep 15 10:44:45.839922 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.839878    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tls-certs\") pod \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") "
Sep 15 10:44:45.840095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.839992    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-home\") pod \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") "
Sep 15 10:44:45.840095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.840058    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-model-cache" (OuterVolumeSpecName: "model-cache") pod "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" (UID: "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.840095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.840071    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89zq7\" (UniqueName: \"kubernetes.io/projected/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-kube-api-access-89zq7\") pod \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\" (UID: \"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0\") "
Sep 15 10:44:45.840513 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.840487    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:45.840650 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.840630    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-home" (OuterVolumeSpecName: "home") pod "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" (UID: "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.841273 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.841239    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" (UID: "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.842746 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.842718    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-kube-api-access-89zq7" (OuterVolumeSpecName: "kube-api-access-89zq7") pod "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" (UID: "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0"). InnerVolumeSpecName "kube-api-access-89zq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:44:45.843256 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.843237    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" (UID: "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:44:45.843561 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.843533    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-dshm" (OuterVolumeSpecName: "dshm") pod "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" (UID: "d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.845265 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.845237    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:44:45.848535 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.848515    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:44:45.851899 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.851883    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:44:45.941106 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941077    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkn8v\" (UniqueName: \"kubernetes.io/projected/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-kube-api-access-rkn8v\") pod \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") "
Sep 15 10:44:45.941262 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941126    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-model-cache\") pod \"699599fa-ce28-4c30-983d-778af87e42f4\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") "
Sep 15 10:44:45.941351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941257    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn7ds\" (UniqueName: \"kubernetes.io/projected/699599fa-ce28-4c30-983d-778af87e42f4-kube-api-access-vn7ds\") pod \"699599fa-ce28-4c30-983d-778af87e42f4\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") "
Sep 15 10:44:45.941413 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941346    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "699599fa-ce28-4c30-983d-778af87e42f4" (UID: "699599fa-ce28-4c30-983d-778af87e42f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.941413 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941351    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tls-certs\") pod \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") "
Sep 15 10:44:45.941413 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941393    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699599fa-ce28-4c30-983d-778af87e42f4-tls-certs\") pod \"699599fa-ce28-4c30-983d-778af87e42f4\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") "
Sep 15 10:44:45.941573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941416    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7r86\" (UniqueName: \"kubernetes.io/projected/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-kube-api-access-r7r86\") pod \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") "
Sep 15 10:44:45.941573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941435    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tls-certs\") pod \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") "
Sep 15 10:44:45.941573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941455    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-model-cache\") pod \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") "
Sep 15 10:44:45.941573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941479    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-dshm\") pod \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") "
Sep 15 10:44:45.941573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941502    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-dshm\") pod \"699599fa-ce28-4c30-983d-778af87e42f4\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") "
Sep 15 10:44:45.941573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941532    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-home\") pod \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") "
Sep 15 10:44:45.941573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941563    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tmp-dir\") pod \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") "
Sep 15 10:44:45.941961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941602    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tmp-dir\") pod \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") "
Sep 15 10:44:45.941961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941686    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-home\") pod \"699599fa-ce28-4c30-983d-778af87e42f4\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") "
Sep 15 10:44:45.941961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941726    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e6a8377-e9ba-4d65-9e4e-e55541d6c288" (UID: "2e6a8377-e9ba-4d65-9e4e-e55541d6c288"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.941961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941751    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-dshm\") pod \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") "
Sep 15 10:44:45.941961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941794    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-home\") pod \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\" (UID: \"2e6a8377-e9ba-4d65-9e4e-e55541d6c288\") "
Sep 15 10:44:45.941961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941839    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-tmp-dir\") pod \"699599fa-ce28-4c30-983d-778af87e42f4\" (UID: \"699599fa-ce28-4c30-983d-778af87e42f4\") "
Sep 15 10:44:45.941961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941864    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-model-cache\") pod \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\" (UID: \"29d873a2-0f82-45a3-9fcf-cf193bde0e8c\") "
Sep 15 10:44:45.941961 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.941949    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-home" (OuterVolumeSpecName: "home") pod "29d873a2-0f82-45a3-9fcf-cf193bde0e8c" (UID: "29d873a2-0f82-45a3-9fcf-cf193bde0e8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.942406 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942249    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89zq7\" (UniqueName: \"kubernetes.io/projected/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-kube-api-access-89zq7\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:45.942406 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942270    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:45.942406 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942284    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:45.942406 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942298    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:45.942406 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942312    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:45.942406 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942325    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:45.942406 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942341    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:45.942406 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942353    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:45.942827 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942408    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-home" (OuterVolumeSpecName: "home") pod "699599fa-ce28-4c30-983d-778af87e42f4" (UID: "699599fa-ce28-4c30-983d-778af87e42f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.942827 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942613    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29d873a2-0f82-45a3-9fcf-cf193bde0e8c" (UID: "29d873a2-0f82-45a3-9fcf-cf193bde0e8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.942935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942786    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e6a8377-e9ba-4d65-9e4e-e55541d6c288" (UID: "2e6a8377-e9ba-4d65-9e4e-e55541d6c288"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.942935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942824    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "29d873a2-0f82-45a3-9fcf-cf193bde0e8c" (UID: "29d873a2-0f82-45a3-9fcf-cf193bde0e8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.942935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.942865    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-home" (OuterVolumeSpecName: "home") pod "2e6a8377-e9ba-4d65-9e4e-e55541d6c288" (UID: "2e6a8377-e9ba-4d65-9e4e-e55541d6c288"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.943099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.943006    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "699599fa-ce28-4c30-983d-778af87e42f4" (UID: "699599fa-ce28-4c30-983d-778af87e42f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.943981 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.943954    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-kube-api-access-r7r86" (OuterVolumeSpecName: "kube-api-access-r7r86") pod "2e6a8377-e9ba-4d65-9e4e-e55541d6c288" (UID: "2e6a8377-e9ba-4d65-9e4e-e55541d6c288"). InnerVolumeSpecName "kube-api-access-r7r86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:44:45.944104 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.944049    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699599fa-ce28-4c30-983d-778af87e42f4-kube-api-access-vn7ds" (OuterVolumeSpecName: "kube-api-access-vn7ds") pod "699599fa-ce28-4c30-983d-778af87e42f4" (UID: "699599fa-ce28-4c30-983d-778af87e42f4"). InnerVolumeSpecName "kube-api-access-vn7ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:44:45.944367 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.944342    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e6a8377-e9ba-4d65-9e4e-e55541d6c288" (UID: "2e6a8377-e9ba-4d65-9e4e-e55541d6c288"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:44:45.944469 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.944424    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-kube-api-access-rkn8v" (OuterVolumeSpecName: "kube-api-access-rkn8v") pod "29d873a2-0f82-45a3-9fcf-cf193bde0e8c" (UID: "29d873a2-0f82-45a3-9fcf-cf193bde0e8c"). InnerVolumeSpecName "kube-api-access-rkn8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:44:45.944625 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.944598    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-dshm" (OuterVolumeSpecName: "dshm") pod "2e6a8377-e9ba-4d65-9e4e-e55541d6c288" (UID: "2e6a8377-e9ba-4d65-9e4e-e55541d6c288"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.944753 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.944732    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29d873a2-0f82-45a3-9fcf-cf193bde0e8c" (UID: "29d873a2-0f82-45a3-9fcf-cf193bde0e8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:44:45.944896 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.944879    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-dshm" (OuterVolumeSpecName: "dshm") pod "699599fa-ce28-4c30-983d-778af87e42f4" (UID: "699599fa-ce28-4c30-983d-778af87e42f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.945251 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.945227    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-dshm" (OuterVolumeSpecName: "dshm") pod "29d873a2-0f82-45a3-9fcf-cf193bde0e8c" (UID: "29d873a2-0f82-45a3-9fcf-cf193bde0e8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:44:45.945524 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.945499    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699599fa-ce28-4c30-983d-778af87e42f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "699599fa-ce28-4c30-983d-778af87e42f4" (UID: "699599fa-ce28-4c30-983d-778af87e42f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:44:45.972076 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.972045    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"]
Sep 15 10:44:45.972296 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:45.972272    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf" podUID="2b5fe564-c4f1-4ad3-8d5d-75d90417cb28" containerName="maas-api" containerID="cri-o://3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891" gracePeriod=30
Sep 15 10:44:46.043255 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043218    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043255 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043257    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043271    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043283    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043295    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043308    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043324    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkn8v\" (UniqueName: \"kubernetes.io/projected/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-kube-api-access-rkn8v\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043336    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn7ds\" (UniqueName: \"kubernetes.io/projected/699599fa-ce28-4c30-983d-778af87e42f4-kube-api-access-vn7ds\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043351    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043365    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699599fa-ce28-4c30-983d-778af87e42f4-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043378    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7r86\" (UniqueName: \"kubernetes.io/projected/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-kube-api-access-r7r86\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043390    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043403    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6a8377-e9ba-4d65-9e4e-e55541d6c288-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043415    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/699599fa-ce28-4c30-983d-778af87e42f4-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.043442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.043425    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d873a2-0f82-45a3-9fcf-cf193bde0e8c-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.208188 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.208165    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:44:46.246272 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.246246    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-openshift-service-ca\") pod \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") "
Sep 15 10:44:46.246442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.246305    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-metrics-certs\") pod \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") "
Sep 15 10:44:46.246442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.246359    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5ppv\" (UniqueName: \"kubernetes.io/projected/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-kube-api-access-z5ppv\") pod \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") "
Sep 15 10:44:46.246442 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.246426    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-maas-api-tls\") pod \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\" (UID: \"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28\") "
Sep 15 10:44:46.246892 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.246866    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b5fe564-c4f1-4ad3-8d5d-75d90417cb28" (UID: "2b5fe564-c4f1-4ad3-8d5d-75d90417cb28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:44:46.248496 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.248474    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2b5fe564-c4f1-4ad3-8d5d-75d90417cb28" (UID: "2b5fe564-c4f1-4ad3-8d5d-75d90417cb28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:44:46.248984 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.248960    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b5fe564-c4f1-4ad3-8d5d-75d90417cb28" (UID: "2b5fe564-c4f1-4ad3-8d5d-75d90417cb28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:44:46.249059 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.248962    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-kube-api-access-z5ppv" (OuterVolumeSpecName: "kube-api-access-z5ppv") pod "2b5fe564-c4f1-4ad3-8d5d-75d90417cb28" (UID: "2b5fe564-c4f1-4ad3-8d5d-75d90417cb28"). InnerVolumeSpecName "kube-api-access-z5ppv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:44:46.347200 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.347163    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-openshift-service-ca\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.347200 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.347192    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-metrics-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.347200 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.347201    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5ppv\" (UniqueName: \"kubernetes.io/projected/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-kube-api-access-z5ppv\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.347200 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.347210    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28-maas-api-tls\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:44:46.351441 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.351407    2584 generic.go:356] "Generic (PLEG): container finished" podID="2e6a8377-e9ba-4d65-9e4e-e55541d6c288" containerID="86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752" exitCode=0
Sep 15 10:44:46.351621 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.351449    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"
Sep 15 10:44:46.351621 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.351501    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt" event={"ID":"2e6a8377-e9ba-4d65-9e4e-e55541d6c288","Type":"ContainerDied","Data":"86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752"}
Sep 15 10:44:46.351621 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.351544    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt" event={"ID":"2e6a8377-e9ba-4d65-9e4e-e55541d6c288","Type":"ContainerDied","Data":"09790ebccebf989a0faec7e6885450bd964117bfcdae76c6a0dfce5c94f80974"}
Sep 15 10:44:46.351621 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.351563    2584 scope.go:117] "RemoveContainer" containerID="86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752"
Sep 15 10:44:46.352919 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.352897    2584 generic.go:356] "Generic (PLEG): container finished" podID="d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" containerID="6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224" exitCode=0
Sep 15 10:44:46.353036 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.352929    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn" event={"ID":"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0","Type":"ContainerDied","Data":"6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224"}
Sep 15 10:44:46.353036 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.352976    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn" event={"ID":"d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0","Type":"ContainerDied","Data":"f7a903fbb0c27cffff7dc0c33a912f6f1f452793f5518c953968f90cc644414e"}
Sep 15 10:44:46.353036 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.352942    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"
Sep 15 10:44:46.354663 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.354642    2584 generic.go:356] "Generic (PLEG): container finished" podID="2b5fe564-c4f1-4ad3-8d5d-75d90417cb28" containerID="3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891" exitCode=0
Sep 15 10:44:46.354766 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.354687    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf" event={"ID":"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28","Type":"ContainerDied","Data":"3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891"}
Sep 15 10:44:46.354766 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.354739    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf" event={"ID":"2b5fe564-c4f1-4ad3-8d5d-75d90417cb28","Type":"ContainerDied","Data":"d096bb55f2da80e6a24a79e8dffc5641bcf5c9da9121bc349eee363edfec52f8"}
Sep 15 10:44:46.354943 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.354693    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"
Sep 15 10:44:46.356310 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.356287    2584 generic.go:356] "Generic (PLEG): container finished" podID="29d873a2-0f82-45a3-9fcf-cf193bde0e8c" containerID="2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002" exitCode=0
Sep 15 10:44:46.356394 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.356326    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"
Sep 15 10:44:46.356394 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.356366    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl" event={"ID":"29d873a2-0f82-45a3-9fcf-cf193bde0e8c","Type":"ContainerDied","Data":"2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002"}
Sep 15 10:44:46.356591 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.356393    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl" event={"ID":"29d873a2-0f82-45a3-9fcf-cf193bde0e8c","Type":"ContainerDied","Data":"aaa460e5315ca8d209354e752ac5b748c4dbf76ba0181c460a4ba62f6f63ed0e"}
Sep 15 10:44:46.360472 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.360080    2584 generic.go:356] "Generic (PLEG): container finished" podID="699599fa-ce28-4c30-983d-778af87e42f4" containerID="9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd" exitCode=0
Sep 15 10:44:46.360472 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.360169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c" event={"ID":"699599fa-ce28-4c30-983d-778af87e42f4","Type":"ContainerDied","Data":"9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd"}
Sep 15 10:44:46.360472 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.360197    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c" event={"ID":"699599fa-ce28-4c30-983d-778af87e42f4","Type":"ContainerDied","Data":"901a99ee18fb8d7d57fddfb6cf7a6e560605567fbf76086f6d9f9355250a4d9c"}
Sep 15 10:44:46.360472 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.360248    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"
Sep 15 10:44:46.363443 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.363418    2584 scope.go:117] "RemoveContainer" containerID="86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752"
Sep 15 10:44:46.363853 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:44:46.363791    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752\": container with ID starting with 86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752 not found: ID does not exist" containerID="86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752"
Sep 15 10:44:46.363950 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.363864    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752"} err="failed to get container status \"86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752\": rpc error: code = NotFound desc = could not find container \"86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752\": container with ID starting with 86992a374e0906397405094e3f0fe6188514b601a05bd40c9b4706322e085752 not found: ID does not exist"
Sep 15 10:44:46.363950 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.363886    2584 scope.go:117] "RemoveContainer" containerID="6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224"
Sep 15 10:44:46.375016 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.374993    2584 scope.go:117] "RemoveContainer" containerID="6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224"
Sep 15 10:44:46.375323 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:44:46.375301    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224\": container with ID starting with 6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224 not found: ID does not exist" containerID="6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224"
Sep 15 10:44:46.375419 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.375333    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224"} err="failed to get container status \"6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224\": rpc error: code = NotFound desc = could not find container \"6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224\": container with ID starting with 6c44c4755d5bf7bc4fb231e0ef74d790510861410f2ce65b1f9ed1ee8d2f7224 not found: ID does not exist"
Sep 15 10:44:46.375419 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.375355    2584 scope.go:117] "RemoveContainer" containerID="3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891"
Sep 15 10:44:46.383585 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.383563    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"]
Sep 15 10:44:46.385465 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.385427    2584 scope.go:117] "RemoveContainer" containerID="3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891"
Sep 15 10:44:46.385844 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:44:46.385822    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891\": container with ID starting with 3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891 not found: ID does not exist" containerID="3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891"
Sep 15 10:44:46.385953 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.385856    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891"} err="failed to get container status \"3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891\": rpc error: code = NotFound desc = could not find container \"3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891\": container with ID starting with 3655c39ac9b7a74d7e2300f1d9807a93799d3f1a10a5d09e099d6446df33c891 not found: ID does not exist"
Sep 15 10:44:46.385953 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.385878    2584 scope.go:117] "RemoveContainer" containerID="2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002"
Sep 15 10:44:46.390243 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.390223    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/distinct-w0-8c1047-kserve-5975f79c8b-9b4mn"]
Sep 15 10:44:46.395243 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.395225    2584 scope.go:117] "RemoveContainer" containerID="2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002"
Sep 15 10:44:46.395513 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:44:46.395493    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002\": container with ID starting with 2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002 not found: ID does not exist" containerID="2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002"
Sep 15 10:44:46.395564 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.395518    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002"} err="failed to get container status \"2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002\": rpc error: code = NotFound desc = could not find container \"2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002\": container with ID starting with 2da2c1a3028f71517ff414ba0b69999e0308e53ae51937d8c52da1227b76f002 not found: ID does not exist"
Sep 15 10:44:46.395564 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.395535    2584 scope.go:117] "RemoveContainer" containerID="9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd"
Sep 15 10:44:46.401638 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.401617    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"]
Sep 15 10:44:46.403494 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.403474    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt"]
Sep 15 10:44:46.405351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.405336    2584 scope.go:117] "RemoveContainer" containerID="9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd"
Sep 15 10:44:46.405608 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:44:46.405590    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd\": container with ID starting with 9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd not found: ID does not exist" containerID="9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd"
Sep 15 10:44:46.405674 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.405612    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd"} err="failed to get container status \"9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd\": rpc error: code = NotFound desc = could not find container \"9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd\": container with ID starting with 9434e8fd253c2923e21d86adbc8d63b3a182f0d3863df330edcb3422e5a923bd not found: ID does not exist"
Sep 15 10:44:46.414471 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.414453    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"]
Sep 15 10:44:46.416071 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.416051    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/premium-w0-8c1047-kserve-666dbd5487-jwtwl"]
Sep 15 10:44:46.423515 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.423495    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"]
Sep 15 10:44:46.425888 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.425867    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8c1047/simulator-w0-8c1047-kserve-86c59487bb-9kf8c"]
Sep 15 10:44:46.435685 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.435667    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"]
Sep 15 10:44:46.439510 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:46.439492    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-30651f-8f7b9cd69-9tdmf"]
Sep 15 10:44:47.786385 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:47.786351    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29d873a2-0f82-45a3-9fcf-cf193bde0e8c" path="/var/lib/kubelet/pods/29d873a2-0f82-45a3-9fcf-cf193bde0e8c/volumes"
Sep 15 10:44:47.786769 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:47.786738    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b5fe564-c4f1-4ad3-8d5d-75d90417cb28" path="/var/lib/kubelet/pods/2b5fe564-c4f1-4ad3-8d5d-75d90417cb28/volumes"
Sep 15 10:44:47.787115 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:47.787102    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6a8377-e9ba-4d65-9e4e-e55541d6c288" path="/var/lib/kubelet/pods/2e6a8377-e9ba-4d65-9e4e-e55541d6c288/volumes"
Sep 15 10:44:47.787749 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:47.787734    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="699599fa-ce28-4c30-983d-778af87e42f4" path="/var/lib/kubelet/pods/699599fa-ce28-4c30-983d-778af87e42f4/volumes"
Sep 15 10:44:47.788146 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:47.788132    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" path="/var/lib/kubelet/pods/d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0/volumes"
Sep 15 10:44:50.774546 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:44:50.774504    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8c1047/embedding-w0-8c1047-kserve-6dfb6498f9-qqmrt" podUID="2e6a8377-e9ba-4d65-9e4e-e55541d6c288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 15 10:45:00.136077 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.135990    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"]
Sep 15 10:45:00.136553 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136534    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29d873a2-0f82-45a3-9fcf-cf193bde0e8c" containerName="main"
Sep 15 10:45:00.136642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136556    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="29d873a2-0f82-45a3-9fcf-cf193bde0e8c" containerName="main"
Sep 15 10:45:00.136642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136569    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b5fe564-c4f1-4ad3-8d5d-75d90417cb28" containerName="maas-api"
Sep 15 10:45:00.136642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136579    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b5fe564-c4f1-4ad3-8d5d-75d90417cb28" containerName="maas-api"
Sep 15 10:45:00.136642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136590    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01aeaff-8342-4127-b7db-ed523673e74c" containerName="payload-processing"
Sep 15 10:45:00.136642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136600    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01aeaff-8342-4127-b7db-ed523673e74c" containerName="payload-processing"
Sep 15 10:45:00.136642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136619    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6a8377-e9ba-4d65-9e4e-e55541d6c288" containerName="main"
Sep 15 10:45:00.136642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136628    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6a8377-e9ba-4d65-9e4e-e55541d6c288" containerName="main"
Sep 15 10:45:00.136642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136639    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" containerName="main"
Sep 15 10:45:00.136642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136647    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" containerName="main"
Sep 15 10:45:00.137099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136664    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62739fd7-9067-46b6-83ee-7b5e398f839a" containerName="payload-pre-processing"
Sep 15 10:45:00.137099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136673    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="62739fd7-9067-46b6-83ee-7b5e398f839a" containerName="payload-pre-processing"
Sep 15 10:45:00.137099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136684    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="699599fa-ce28-4c30-983d-778af87e42f4" containerName="main"
Sep 15 10:45:00.137099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136692    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="699599fa-ce28-4c30-983d-778af87e42f4" containerName="main"
Sep 15 10:45:00.137099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136790    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="62739fd7-9067-46b6-83ee-7b5e398f839a" containerName="payload-pre-processing"
Sep 15 10:45:00.137099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136828    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b5fe564-c4f1-4ad3-8d5d-75d90417cb28" containerName="maas-api"
Sep 15 10:45:00.137099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136838    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6a8377-e9ba-4d65-9e4e-e55541d6c288" containerName="main"
Sep 15 10:45:00.137099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136850    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e01aeaff-8342-4127-b7db-ed523673e74c" containerName="payload-processing"
Sep 15 10:45:00.137099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136860    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="699599fa-ce28-4c30-983d-778af87e42f4" containerName="main"
Sep 15 10:45:00.137099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136869    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="29d873a2-0f82-45a3-9fcf-cf193bde0e8c" containerName="main"
Sep 15 10:45:00.137099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.136878    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d40ebc9b-bf8f-49fb-b6ed-95fa452f7aa0" containerName="main"
Sep 15 10:45:00.141976 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.141948    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c"]
Sep 15 10:45:00.142122 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.142062    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"
Sep 15 10:45:00.144340 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.144313    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jrzm\""
Sep 15 10:45:00.146090 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.146063    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c"]
Sep 15 10:45:00.146219 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.146172    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c"
Sep 15 10:45:00.148730 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.148702    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"]
Sep 15 10:45:00.157897 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.157876    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"
Sep 15 10:45:00.165158 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.165135    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c"
Sep 15 10:45:00.430956 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.430924    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"]
Sep 15 10:45:00.445545 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:00.445515    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c"]
Sep 15 10:45:00.448047 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:45:00.448017    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7d7fa2_9207_47c7_bb26_9c1fa07ef130.slice/crio-186fc7a7dd26b3721d89aa5ac5d79b4575f2b7adc4949379efb4ce6ea85bd076 WatchSource:0}: Error finding container 186fc7a7dd26b3721d89aa5ac5d79b4575f2b7adc4949379efb4ce6ea85bd076: Status 404 returned error can't find the container with id 186fc7a7dd26b3721d89aa5ac5d79b4575f2b7adc4949379efb4ce6ea85bd076
Sep 15 10:45:01.446046 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:01.445954    2584 generic.go:356] "Generic (PLEG): container finished" podID="6d3d13f0-5495-4121-956e-d81704445090" containerID="bdd6285e305ee8f97525bda580421b3dd09998cc34575cf17957be119774029c" exitCode=0
Sep 15 10:45:01.446480 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:01.446045    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj" event={"ID":"6d3d13f0-5495-4121-956e-d81704445090","Type":"ContainerDied","Data":"bdd6285e305ee8f97525bda580421b3dd09998cc34575cf17957be119774029c"}
Sep 15 10:45:01.446480 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:01.446103    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj" event={"ID":"6d3d13f0-5495-4121-956e-d81704445090","Type":"ContainerStarted","Data":"606e582e0d31fe2896ebc97afb5eecf83313482073a5d0a8fc455e96942f3816"}
Sep 15 10:45:01.447625 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:01.447604    2584 generic.go:356] "Generic (PLEG): container finished" podID="de7d7fa2-9207-47c7-bb26-9c1fa07ef130" containerID="2a4fc88e7ae29c4b6cfe25d3064b4def716ef93ceb7e0db60a04e4ee4f8f5b2c" exitCode=0
Sep 15 10:45:01.447729 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:01.447676    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c" event={"ID":"de7d7fa2-9207-47c7-bb26-9c1fa07ef130","Type":"ContainerDied","Data":"2a4fc88e7ae29c4b6cfe25d3064b4def716ef93ceb7e0db60a04e4ee4f8f5b2c"}
Sep 15 10:45:01.447783 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:01.447730    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c" event={"ID":"de7d7fa2-9207-47c7-bb26-9c1fa07ef130","Type":"ContainerStarted","Data":"186fc7a7dd26b3721d89aa5ac5d79b4575f2b7adc4949379efb4ce6ea85bd076"}
Sep 15 10:45:02.606241 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:02.606215    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"
Sep 15 10:45:02.626543 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:02.626516    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c"
Sep 15 10:45:03.454387 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.454348    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"]
Sep 15 10:45:03.458820 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.458773    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"
Sep 15 10:45:03.458820 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.458784    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj" event={"ID":"6d3d13f0-5495-4121-956e-d81704445090","Type":"ContainerDied","Data":"606e582e0d31fe2896ebc97afb5eecf83313482073a5d0a8fc455e96942f3816"}
Sep 15 10:45:03.459036 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.458844    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="606e582e0d31fe2896ebc97afb5eecf83313482073a5d0a8fc455e96942f3816"
Sep 15 10:45:03.460282 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.460261    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c"
Sep 15 10:45:03.460422 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.460285    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c" event={"ID":"de7d7fa2-9207-47c7-bb26-9c1fa07ef130","Type":"ContainerDied","Data":"186fc7a7dd26b3721d89aa5ac5d79b4575f2b7adc4949379efb4ce6ea85bd076"}
Sep 15 10:45:03.460422 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.460321    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="186fc7a7dd26b3721d89aa5ac5d79b4575f2b7adc4949379efb4ce6ea85bd076"
Sep 15 10:45:03.464034 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.464008    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"]
Sep 15 10:45:03.473453 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:45:03.473189    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj\" is forbidden: User \"system:node:ip-10-0-132-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-132-90.ec2.internal' and this object" podUID="6d3d13f0-5495-4121-956e-d81704445090" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"
Sep 15 10:45:03.474922 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:45:03.474885    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj\" is forbidden: User \"system:node:ip-10-0-132-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-132-90.ec2.internal' and this object" podUID="6d3d13f0-5495-4121-956e-d81704445090" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"
Sep 15 10:45:03.483073 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.483044    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"]
Sep 15 10:45:03.483271 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.483245    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj" podUID="5a6f2956-fd66-4701-be8b-696e178d230e" containerName="payload-pre-processing" containerID="cri-o://0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247" gracePeriod=30
Sep 15 10:45:03.485416 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:45:03.485262    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj\" is forbidden: User \"system:node:ip-10-0-132-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-132-90.ec2.internal' and this object" podUID="6d3d13f0-5495-4121-956e-d81704445090" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"
Sep 15 10:45:03.742366 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.742341    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:45:03.744383 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:45:03.744353    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj\" is forbidden: User \"system:node:ip-10-0-132-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-132-90.ec2.internal' and this object" podUID="6d3d13f0-5495-4121-956e-d81704445090" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ecf56f-29824485-8r8xj"
Sep 15 10:45:03.785296 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.785250    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d3d13f0-5495-4121-956e-d81704445090" path="/var/lib/kubelet/pods/6d3d13f0-5495-4121-956e-d81704445090/volumes"
Sep 15 10:45:03.818713 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.818677    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w67p7\" (UniqueName: \"kubernetes.io/projected/5a6f2956-fd66-4701-be8b-696e178d230e-kube-api-access-w67p7\") pod \"5a6f2956-fd66-4701-be8b-696e178d230e\" (UID: \"5a6f2956-fd66-4701-be8b-696e178d230e\") "
Sep 15 10:45:03.818885 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.818757    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6f2956-fd66-4701-be8b-696e178d230e-plugins-config-volume\") pod \"5a6f2956-fd66-4701-be8b-696e178d230e\" (UID: \"5a6f2956-fd66-4701-be8b-696e178d230e\") "
Sep 15 10:45:03.819121 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.819098    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6f2956-fd66-4701-be8b-696e178d230e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a6f2956-fd66-4701-be8b-696e178d230e" (UID: "5a6f2956-fd66-4701-be8b-696e178d230e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:45:03.820793 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.820772    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6f2956-fd66-4701-be8b-696e178d230e-kube-api-access-w67p7" (OuterVolumeSpecName: "kube-api-access-w67p7") pod "5a6f2956-fd66-4701-be8b-696e178d230e" (UID: "5a6f2956-fd66-4701-be8b-696e178d230e"). InnerVolumeSpecName "kube-api-access-w67p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:45:03.920259 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.920224    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w67p7\" (UniqueName: \"kubernetes.io/projected/5a6f2956-fd66-4701-be8b-696e178d230e-kube-api-access-w67p7\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:45:03.920259 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:03.920257    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6f2956-fd66-4701-be8b-696e178d230e-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:45:04.466584 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:04.466540    2584 generic.go:356] "Generic (PLEG): container finished" podID="5a6f2956-fd66-4701-be8b-696e178d230e" containerID="0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247" exitCode=0
Sep 15 10:45:04.466774 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:04.466599    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"
Sep 15 10:45:04.466774 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:04.466630    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj" event={"ID":"5a6f2956-fd66-4701-be8b-696e178d230e","Type":"ContainerDied","Data":"0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247"}
Sep 15 10:45:04.466774 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:04.466685    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj" event={"ID":"5a6f2956-fd66-4701-be8b-696e178d230e","Type":"ContainerDied","Data":"9fd0039851597111586b1524a13e5d2c9713c14df2e3d97b5e192ad98da7f8b9"}
Sep 15 10:45:04.466774 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:04.466707    2584 scope.go:117] "RemoveContainer" containerID="0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247"
Sep 15 10:45:04.475761 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:04.475736    2584 scope.go:117] "RemoveContainer" containerID="0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247"
Sep 15 10:45:04.476102 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:45:04.476079    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247\": container with ID starting with 0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247 not found: ID does not exist" containerID="0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247"
Sep 15 10:45:04.476169 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:04.476109    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247"} err="failed to get container status \"0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247\": rpc error: code = NotFound desc = could not find container \"0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247\": container with ID starting with 0be0e0c4cb92c5108f97289dff613dd6cadddc5521533c5c3c626c5c124a3247 not found: ID does not exist"
Sep 15 10:45:04.486767 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:04.486739    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"]
Sep 15 10:45:04.490360 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:04.490336    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ecf56f-987d6bdbb-wkvrj"]
Sep 15 10:45:05.787161 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:05.787131    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a6f2956-fd66-4701-be8b-696e178d230e" path="/var/lib/kubelet/pods/5a6f2956-fd66-4701-be8b-696e178d230e/volumes"
Sep 15 10:45:07.760012 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:07.759976    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:45:30.550973 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:30.550938    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:45:51.449456 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:45:51.449419    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:46:12.552866 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:12.552815    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:46:22.891573 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:22.891537    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"]
Sep 15 10:46:22.897555 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:22.897519    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"]
Sep 15 10:46:22.910060 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:22.910026    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"]
Sep 15 10:46:22.918656 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:22.918622    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"]
Sep 15 10:46:33.966614 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:33.966526    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:46:37.912732 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:37.912649    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n" podUID="9284f93f-ee7f-4d43-879d-a169df42db13" containerName="main" containerID="cri-o://bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8" gracePeriod=45
Sep 15 10:46:37.920417 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:37.920352    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj" podUID="7cc17d65-a9c9-4e35-b2d8-41dddfd31759" containerName="main" containerID="cri-o://ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21" gracePeriod=45
Sep 15 10:46:37.929518 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:37.929454    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt" podUID="e3798e11-c021-4603-98d5-2c940a59df7e" containerName="main" containerID="cri-o://f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a" gracePeriod=45
Sep 15 10:46:37.942545 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:37.942517    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw" podUID="bb3153b5-85b6-4565-b625-b9ab7092c37d" containerName="main" containerID="cri-o://d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e" gracePeriod=45
Sep 15 10:46:38.214110 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.214079    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:46:38.240717 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.240680    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:46:38.250865 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.250841    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:46:38.254351 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.254331    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:46:38.324609 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324575    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-model-cache\") pod \"bb3153b5-85b6-4565-b625-b9ab7092c37d\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") "
Sep 15 10:46:38.324790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324622    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-model-cache\") pod \"9284f93f-ee7f-4d43-879d-a169df42db13\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") "
Sep 15 10:46:38.324790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324655    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-dshm\") pod \"bb3153b5-85b6-4565-b625-b9ab7092c37d\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") "
Sep 15 10:46:38.324790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324679    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-dshm\") pod \"e3798e11-c021-4603-98d5-2c940a59df7e\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") "
Sep 15 10:46:38.324790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324710    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-tmp-dir\") pod \"e3798e11-c021-4603-98d5-2c940a59df7e\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") "
Sep 15 10:46:38.324790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324751    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-home\") pod \"bb3153b5-85b6-4565-b625-b9ab7092c37d\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") "
Sep 15 10:46:38.324790 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324776    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-model-cache\") pod \"e3798e11-c021-4603-98d5-2c940a59df7e\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") "
Sep 15 10:46:38.325069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324821    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-model-cache\") pod \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") "
Sep 15 10:46:38.325069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324846    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-tmp-dir\") pod \"9284f93f-ee7f-4d43-879d-a169df42db13\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") "
Sep 15 10:46:38.325069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324866    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb3153b5-85b6-4565-b625-b9ab7092c37d" (UID: "bb3153b5-85b6-4565-b625-b9ab7092c37d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.325069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324873    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-tmp-dir\") pod \"bb3153b5-85b6-4565-b625-b9ab7092c37d\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") "
Sep 15 10:46:38.325069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324937    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-home\") pod \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") "
Sep 15 10:46:38.325069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.324977    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28lsg\" (UniqueName: \"kubernetes.io/projected/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-kube-api-access-28lsg\") pod \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") "
Sep 15 10:46:38.325069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325017    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tmp-dir\") pod \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") "
Sep 15 10:46:38.325069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325070    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-dshm\") pod \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") "
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325075    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb3153b5-85b6-4565-b625-b9ab7092c37d" (UID: "bb3153b5-85b6-4565-b625-b9ab7092c37d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325095    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tls-certs\") pod \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\" (UID: \"7cc17d65-a9c9-4e35-b2d8-41dddfd31759\") "
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325123    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-home" (OuterVolumeSpecName: "home") pod "bb3153b5-85b6-4565-b625-b9ab7092c37d" (UID: "bb3153b5-85b6-4565-b625-b9ab7092c37d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325131    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkzhn\" (UniqueName: \"kubernetes.io/projected/9284f93f-ee7f-4d43-879d-a169df42db13-kube-api-access-dkzhn\") pod \"9284f93f-ee7f-4d43-879d-a169df42db13\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") "
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325188    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb3153b5-85b6-4565-b625-b9ab7092c37d-tls-certs\") pod \"bb3153b5-85b6-4565-b625-b9ab7092c37d\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") "
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325233    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3798e11-c021-4603-98d5-2c940a59df7e" (UID: "e3798e11-c021-4603-98d5-2c940a59df7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325248    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9284f93f-ee7f-4d43-879d-a169df42db13-tls-certs\") pod \"9284f93f-ee7f-4d43-879d-a169df42db13\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") "
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325270    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9284f93f-ee7f-4d43-879d-a169df42db13" (UID: "9284f93f-ee7f-4d43-879d-a169df42db13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325305    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-home\") pod \"9284f93f-ee7f-4d43-879d-a169df42db13\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") "
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325337    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3798e11-c021-4603-98d5-2c940a59df7e-tls-certs\") pod \"e3798e11-c021-4603-98d5-2c940a59df7e\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") "
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325384    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-dshm\") pod \"9284f93f-ee7f-4d43-879d-a169df42db13\" (UID: \"9284f93f-ee7f-4d43-879d-a169df42db13\") "
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325409    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-model-cache" (OuterVolumeSpecName: "model-cache") pod "7cc17d65-a9c9-4e35-b2d8-41dddfd31759" (UID: "7cc17d65-a9c9-4e35-b2d8-41dddfd31759"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325417    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpzw6\" (UniqueName: \"kubernetes.io/projected/bb3153b5-85b6-4565-b625-b9ab7092c37d-kube-api-access-xpzw6\") pod \"bb3153b5-85b6-4565-b625-b9ab7092c37d\" (UID: \"bb3153b5-85b6-4565-b625-b9ab7092c37d\") "
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325445    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3798e11-c021-4603-98d5-2c940a59df7e" (UID: "e3798e11-c021-4603-98d5-2c940a59df7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.325487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325476    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5m79\" (UniqueName: \"kubernetes.io/projected/e3798e11-c021-4603-98d5-2c940a59df7e-kube-api-access-j5m79\") pod \"e3798e11-c021-4603-98d5-2c940a59df7e\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") "
Sep 15 10:46:38.326323 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325526    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-home\") pod \"e3798e11-c021-4603-98d5-2c940a59df7e\" (UID: \"e3798e11-c021-4603-98d5-2c940a59df7e\") "
Sep 15 10:46:38.326323 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325692    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-model-cache" (OuterVolumeSpecName: "model-cache") pod "9284f93f-ee7f-4d43-879d-a169df42db13" (UID: "9284f93f-ee7f-4d43-879d-a169df42db13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.326323 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.325944    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-home" (OuterVolumeSpecName: "home") pod "9284f93f-ee7f-4d43-879d-a169df42db13" (UID: "9284f93f-ee7f-4d43-879d-a169df42db13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.326323 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.326034    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7cc17d65-a9c9-4e35-b2d8-41dddfd31759" (UID: "7cc17d65-a9c9-4e35-b2d8-41dddfd31759"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.326323 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.326233    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-home" (OuterVolumeSpecName: "home") pod "7cc17d65-a9c9-4e35-b2d8-41dddfd31759" (UID: "7cc17d65-a9c9-4e35-b2d8-41dddfd31759"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.327561 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.327531    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-dshm" (OuterVolumeSpecName: "dshm") pod "e3798e11-c021-4603-98d5-2c940a59df7e" (UID: "e3798e11-c021-4603-98d5-2c940a59df7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.327784 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.327765    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-home" (OuterVolumeSpecName: "home") pod "e3798e11-c021-4603-98d5-2c940a59df7e" (UID: "e3798e11-c021-4603-98d5-2c940a59df7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.327935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.327908    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.327935 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.327931    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.328053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.327945    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.328053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.327957    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.328053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.327969    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.328053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.327980    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.328053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.327992    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.328053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.328003    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.328053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.328016    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.328053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.328029    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.328053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.328041    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.328053 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.328052    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.328614 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.328108    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9284f93f-ee7f-4d43-879d-a169df42db13-kube-api-access-dkzhn" (OuterVolumeSpecName: "kube-api-access-dkzhn") pod "9284f93f-ee7f-4d43-879d-a169df42db13" (UID: "9284f93f-ee7f-4d43-879d-a169df42db13"). InnerVolumeSpecName "kube-api-access-dkzhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:46:38.328614 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.328239    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-dshm" (OuterVolumeSpecName: "dshm") pod "bb3153b5-85b6-4565-b625-b9ab7092c37d" (UID: "bb3153b5-85b6-4565-b625-b9ab7092c37d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.328844 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.328789    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-dshm" (OuterVolumeSpecName: "dshm") pod "7cc17d65-a9c9-4e35-b2d8-41dddfd31759" (UID: "7cc17d65-a9c9-4e35-b2d8-41dddfd31759"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.328988 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.328962    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3153b5-85b6-4565-b625-b9ab7092c37d-kube-api-access-xpzw6" (OuterVolumeSpecName: "kube-api-access-xpzw6") pod "bb3153b5-85b6-4565-b625-b9ab7092c37d" (UID: "bb3153b5-85b6-4565-b625-b9ab7092c37d"). InnerVolumeSpecName "kube-api-access-xpzw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:46:38.329278 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.329243    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3153b5-85b6-4565-b625-b9ab7092c37d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb3153b5-85b6-4565-b625-b9ab7092c37d" (UID: "bb3153b5-85b6-4565-b625-b9ab7092c37d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:46:38.329403 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.329315    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-kube-api-access-28lsg" (OuterVolumeSpecName: "kube-api-access-28lsg") pod "7cc17d65-a9c9-4e35-b2d8-41dddfd31759" (UID: "7cc17d65-a9c9-4e35-b2d8-41dddfd31759"). InnerVolumeSpecName "kube-api-access-28lsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:46:38.330165 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.330139    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3798e11-c021-4603-98d5-2c940a59df7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3798e11-c021-4603-98d5-2c940a59df7e" (UID: "e3798e11-c021-4603-98d5-2c940a59df7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:46:38.330288 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.330269    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3798e11-c021-4603-98d5-2c940a59df7e-kube-api-access-j5m79" (OuterVolumeSpecName: "kube-api-access-j5m79") pod "e3798e11-c021-4603-98d5-2c940a59df7e" (UID: "e3798e11-c021-4603-98d5-2c940a59df7e"). InnerVolumeSpecName "kube-api-access-j5m79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:46:38.330361 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.330272    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-dshm" (OuterVolumeSpecName: "dshm") pod "9284f93f-ee7f-4d43-879d-a169df42db13" (UID: "9284f93f-ee7f-4d43-879d-a169df42db13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:46:38.330575 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.330553    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7cc17d65-a9c9-4e35-b2d8-41dddfd31759" (UID: "7cc17d65-a9c9-4e35-b2d8-41dddfd31759"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:46:38.331124 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.331108    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9284f93f-ee7f-4d43-879d-a169df42db13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9284f93f-ee7f-4d43-879d-a169df42db13" (UID: "9284f93f-ee7f-4d43-879d-a169df42db13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:46:38.429100 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429058    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9284f93f-ee7f-4d43-879d-a169df42db13-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.429100 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429097    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpzw6\" (UniqueName: \"kubernetes.io/projected/bb3153b5-85b6-4565-b625-b9ab7092c37d-kube-api-access-xpzw6\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.429100 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429108    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5m79\" (UniqueName: \"kubernetes.io/projected/e3798e11-c021-4603-98d5-2c940a59df7e-kube-api-access-j5m79\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.429345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429118    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3798e11-c021-4603-98d5-2c940a59df7e-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.429345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429127    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb3153b5-85b6-4565-b625-b9ab7092c37d-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.429345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429134    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28lsg\" (UniqueName: \"kubernetes.io/projected/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-kube-api-access-28lsg\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.429345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429142    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.429345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429150    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cc17d65-a9c9-4e35-b2d8-41dddfd31759-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.429345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429163    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkzhn\" (UniqueName: \"kubernetes.io/projected/9284f93f-ee7f-4d43-879d-a169df42db13-kube-api-access-dkzhn\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.429345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429175    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb3153b5-85b6-4565-b625-b9ab7092c37d-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.429345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429184    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9284f93f-ee7f-4d43-879d-a169df42db13-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.429345 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.429191    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3798e11-c021-4603-98d5-2c940a59df7e-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:46:38.924894 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.924859    2584 generic.go:356] "Generic (PLEG): container finished" podID="e3798e11-c021-4603-98d5-2c940a59df7e" containerID="f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a" exitCode=0
Sep 15 10:46:38.925310 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.924906    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"
Sep 15 10:46:38.925310 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.924954    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt" event={"ID":"e3798e11-c021-4603-98d5-2c940a59df7e","Type":"ContainerDied","Data":"f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a"}
Sep 15 10:46:38.925310 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.925003    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt" event={"ID":"e3798e11-c021-4603-98d5-2c940a59df7e","Type":"ContainerDied","Data":"c5c3fcfbf5df3934f37d32c1e7959602626c5cda6527fdb79e5249ecacb38680"}
Sep 15 10:46:38.925310 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.925023    2584 scope.go:117] "RemoveContainer" containerID="f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a"
Sep 15 10:46:38.926812 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.926654    2584 generic.go:356] "Generic (PLEG): container finished" podID="bb3153b5-85b6-4565-b625-b9ab7092c37d" containerID="d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e" exitCode=0
Sep 15 10:46:38.926812 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.926688    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw" event={"ID":"bb3153b5-85b6-4565-b625-b9ab7092c37d","Type":"ContainerDied","Data":"d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e"}
Sep 15 10:46:38.926812 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.926713    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"
Sep 15 10:46:38.926812 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.926729    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw" event={"ID":"bb3153b5-85b6-4565-b625-b9ab7092c37d","Type":"ContainerDied","Data":"16793abe951177d0fff6eb6c1fb5b6dae3e761644660d176499d195e685447f4"}
Sep 15 10:46:38.928627 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.928600    2584 generic.go:356] "Generic (PLEG): container finished" podID="9284f93f-ee7f-4d43-879d-a169df42db13" containerID="bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8" exitCode=0
Sep 15 10:46:38.928968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.928694    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n" event={"ID":"9284f93f-ee7f-4d43-879d-a169df42db13","Type":"ContainerDied","Data":"bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8"}
Sep 15 10:46:38.928968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.928722    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n" event={"ID":"9284f93f-ee7f-4d43-879d-a169df42db13","Type":"ContainerDied","Data":"eeff2c19c2ac5f11911ad48c4fd4867e45ce119f6defab3e94b17db4b525f7a0"}
Sep 15 10:46:38.928968 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.928782    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"
Sep 15 10:46:38.931956 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.931931    2584 generic.go:356] "Generic (PLEG): container finished" podID="7cc17d65-a9c9-4e35-b2d8-41dddfd31759" containerID="ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21" exitCode=0
Sep 15 10:46:38.932061 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.931975    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"
Sep 15 10:46:38.932061 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.931970    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj" event={"ID":"7cc17d65-a9c9-4e35-b2d8-41dddfd31759","Type":"ContainerDied","Data":"ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21"}
Sep 15 10:46:38.932061 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.932024    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj" event={"ID":"7cc17d65-a9c9-4e35-b2d8-41dddfd31759","Type":"ContainerDied","Data":"90b96026c925a85d026e929c03b07cf3e2d0a0cac68bf514ca7b87f3969a8279"}
Sep 15 10:46:38.948781 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.948763    2584 scope.go:117] "RemoveContainer" containerID="f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a"
Sep 15 10:46:38.949165 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:46:38.949145    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a\": container with ID starting with f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a not found: ID does not exist" containerID="f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a"
Sep 15 10:46:38.949251 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.949174    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a"} err="failed to get container status \"f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a\": rpc error: code = NotFound desc = could not find container \"f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a\": container with ID starting with f32fc12bcd8f01ae7533bc3efc91beda87574368b65ae799d747aefcc9ab407a not found: ID does not exist"
Sep 15 10:46:38.949251 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.949190    2584 scope.go:117] "RemoveContainer" containerID="d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e"
Sep 15 10:46:38.957274 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.957255    2584 scope.go:117] "RemoveContainer" containerID="d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e"
Sep 15 10:46:38.957545 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:46:38.957526    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e\": container with ID starting with d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e not found: ID does not exist" containerID="d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e"
Sep 15 10:46:38.957588 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.957552    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e"} err="failed to get container status \"d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e\": rpc error: code = NotFound desc = could not find container \"d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e\": container with ID starting with d47094024894b8ea1d8f065aef6a53fe0f7ae05bde9d083575878d319005f10e not found: ID does not exist"
Sep 15 10:46:38.957588 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.957568    2584 scope.go:117] "RemoveContainer" containerID="bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8"
Sep 15 10:46:38.963454 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.963433    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"]
Sep 15 10:46:38.965205 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.965186    2584 scope.go:117] "RemoveContainer" containerID="bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8"
Sep 15 10:46:38.965615 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:46:38.965590    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8\": container with ID starting with bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8 not found: ID does not exist" containerID="bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8"
Sep 15 10:46:38.965709 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.965620    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8"} err="failed to get container status \"bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8\": rpc error: code = NotFound desc = could not find container \"bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8\": container with ID starting with bef1f6a488de11f5021c5c7a351cc4168250f6551f66bc282119eed60181c6c8 not found: ID does not exist"
Sep 15 10:46:38.965709 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.965637    2584 scope.go:117] "RemoveContainer" containerID="ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21"
Sep 15 10:46:38.967399 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.967377    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/premium-w4-ff0655-kserve-6567f67c6d-j4nwt"]
Sep 15 10:46:38.972857 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.972840    2584 scope.go:117] "RemoveContainer" containerID="ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21"
Sep 15 10:46:38.973123 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:46:38.973105    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21\": container with ID starting with ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21 not found: ID does not exist" containerID="ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21"
Sep 15 10:46:38.973179 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.973128    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21"} err="failed to get container status \"ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21\": rpc error: code = NotFound desc = could not find container \"ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21\": container with ID starting with ad187a9af3567eb681bfa57d28dc52adf6ef9c97e52cf790acec51ad10815f21 not found: ID does not exist"
Sep 15 10:46:38.976537 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.976515    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"]
Sep 15 10:46:38.981777 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.981755    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-2-w4-ff0655-kserve-64c6bdc8c6-tj49n"]
Sep 15 10:46:38.990744 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.990718    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"]
Sep 15 10:46:38.993951 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:38.993927    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/distinct-w4-ff0655-kserve-7b4b4bffff-x7pwj"]
Sep 15 10:46:39.003389 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:39.003365    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"]
Sep 15 10:46:39.006882 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:39.006863    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ff0655/simulator-w4-ff0655-kserve-5554db4bbf-jjbbw"]
Sep 15 10:46:39.784752 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:39.784718    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc17d65-a9c9-4e35-b2d8-41dddfd31759" path="/var/lib/kubelet/pods/7cc17d65-a9c9-4e35-b2d8-41dddfd31759/volumes"
Sep 15 10:46:39.785169 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:39.785155    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9284f93f-ee7f-4d43-879d-a169df42db13" path="/var/lib/kubelet/pods/9284f93f-ee7f-4d43-879d-a169df42db13/volumes"
Sep 15 10:46:39.785506 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:39.785495    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb3153b5-85b6-4565-b625-b9ab7092c37d" path="/var/lib/kubelet/pods/bb3153b5-85b6-4565-b625-b9ab7092c37d/volumes"
Sep 15 10:46:39.786174 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:39.786158    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3798e11-c021-4603-98d5-2c940a59df7e" path="/var/lib/kubelet/pods/e3798e11-c021-4603-98d5-2c940a59df7e/volumes"
Sep 15 10:46:49.953307 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:49.953268    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:46:56.460458 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460423    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"]
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460758    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3153b5-85b6-4565-b625-b9ab7092c37d" containerName="main"
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460769    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3153b5-85b6-4565-b625-b9ab7092c37d" containerName="main"
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460786    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc17d65-a9c9-4e35-b2d8-41dddfd31759" containerName="main"
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460792    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc17d65-a9c9-4e35-b2d8-41dddfd31759" containerName="main"
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460835    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6f2956-fd66-4701-be8b-696e178d230e" containerName="payload-pre-processing"
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460842    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6f2956-fd66-4701-be8b-696e178d230e" containerName="payload-pre-processing"
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460847    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9284f93f-ee7f-4d43-879d-a169df42db13" containerName="main"
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460852    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9284f93f-ee7f-4d43-879d-a169df42db13" containerName="main"
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460861    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3798e11-c021-4603-98d5-2c940a59df7e" containerName="main"
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460867    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3798e11-c021-4603-98d5-2c940a59df7e" containerName="main"
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460873    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3d13f0-5495-4121-956e-d81704445090" containerName="cleanup"
Sep 15 10:46:56.460878 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460878    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3d13f0-5495-4121-956e-d81704445090" containerName="cleanup"
Sep 15 10:46:56.461358 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460887    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de7d7fa2-9207-47c7-bb26-9c1fa07ef130" containerName="cleanup"
Sep 15 10:46:56.461358 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460894    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="de7d7fa2-9207-47c7-bb26-9c1fa07ef130" containerName="cleanup"
Sep 15 10:46:56.461358 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460943    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="de7d7fa2-9207-47c7-bb26-9c1fa07ef130" containerName="cleanup"
Sep 15 10:46:56.461358 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460952    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3798e11-c021-4603-98d5-2c940a59df7e" containerName="main"
Sep 15 10:46:56.461358 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460959    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3d13f0-5495-4121-956e-d81704445090" containerName="cleanup"
Sep 15 10:46:56.461358 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460964    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3153b5-85b6-4565-b625-b9ab7092c37d" containerName="main"
Sep 15 10:46:56.461358 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460970    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc17d65-a9c9-4e35-b2d8-41dddfd31759" containerName="main"
Sep 15 10:46:56.461358 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460975    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a6f2956-fd66-4701-be8b-696e178d230e" containerName="payload-pre-processing"
Sep 15 10:46:56.461358 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.460982    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9284f93f-ee7f-4d43-879d-a169df42db13" containerName="main"
Sep 15 10:46:56.466067 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.466051    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.468399 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.468378    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-83e814\"/\"openshift-service-ca.crt\""
Sep 15 10:46:56.468514 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.468380    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-83e814\"/\"kube-root-ca.crt\""
Sep 15 10:46:56.469061 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.469041    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-83e814\"/\"default-dockercfg-84k6n\""
Sep 15 10:46:56.469161 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.469084    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-83e814\"/\"sub-test-model-83e814-kserve-self-signed-certs\""
Sep 15 10:46:56.473523 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.473429    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"]
Sep 15 10:46:56.599857 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.599793    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e629a6b1-352b-4b37-b56d-e2c104be6016-tls-certs\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.600047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.599926    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-home\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.600047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.599971    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-dshm\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.600154 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.600043    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-model-cache\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.600154 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.600076    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mcr7\" (UniqueName: \"kubernetes.io/projected/e629a6b1-352b-4b37-b56d-e2c104be6016-kube-api-access-9mcr7\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.600154 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.600104    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-tmp-dir\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.701521 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.701478    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-tmp-dir\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.701733 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.701533    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e629a6b1-352b-4b37-b56d-e2c104be6016-tls-certs\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.701733 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.701580    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-home\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.701733 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.701604    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-dshm\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.701951 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.701735    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-model-cache\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.701951 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.701782    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mcr7\" (UniqueName: \"kubernetes.io/projected/e629a6b1-352b-4b37-b56d-e2c104be6016-kube-api-access-9mcr7\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.702057 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.702003    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-tmp-dir\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.702100 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.702073    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-home\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.702148 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.702130    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-model-cache\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.703966 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.703946    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-dshm\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.704260 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.704239    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e629a6b1-352b-4b37-b56d-e2c104be6016-tls-certs\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.709481 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.709457    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mcr7\" (UniqueName: \"kubernetes.io/projected/e629a6b1-352b-4b37-b56d-e2c104be6016-kube-api-access-9mcr7\") pod \"sub-test-model-83e814-kserve-65874dd4df-6wj9v\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") " pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:56.775953 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:56.775919    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:46:57.047402 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:57.047318    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"]
Sep 15 10:46:57.050507 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:46:57.050480    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode629a6b1_352b_4b37_b56d_e2c104be6016.slice/crio-180565fe2b512af0db73138b871566e8fed1d891995d7120a2720097ec823270 WatchSource:0}: Error finding container 180565fe2b512af0db73138b871566e8fed1d891995d7120a2720097ec823270: Status 404 returned error can't find the container with id 180565fe2b512af0db73138b871566e8fed1d891995d7120a2720097ec823270
Sep 15 10:46:58.025417 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:58.025377    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v" event={"ID":"e629a6b1-352b-4b37-b56d-e2c104be6016","Type":"ContainerStarted","Data":"f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5"}
Sep 15 10:46:58.025417 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:58.025420    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v" event={"ID":"e629a6b1-352b-4b37-b56d-e2c104be6016","Type":"ContainerStarted","Data":"180565fe2b512af0db73138b871566e8fed1d891995d7120a2720097ec823270"}
Sep 15 10:46:58.042199 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:46:58.042046    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v" podStartSLOduration=2.042027049 podStartE2EDuration="2.042027049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:46:58.041373496 +0000 UTC m=+1736.735529178" watchObservedRunningTime="2026-09-15 10:46:58.042027049 +0000 UTC m=+1736.736182730"
Sep 15 10:47:06.776462 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:06.776428    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:47:06.776956 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:06.776661    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:47:06.788791 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:06.788766    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:47:07.077201 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:07.077113    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:47:09.558964 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:09.558927    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:47:35.655419 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.655386    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"]
Sep 15 10:47:35.658612 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.658590    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.661918 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.661877    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-44134e\"/\"openshift-service-ca.crt\""
Sep 15 10:47:35.661918 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.661880    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-44134e\"/\"kube-root-ca.crt\""
Sep 15 10:47:35.662120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.661916    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-44134e\"/\"sub-test-model-44134e-kserve-self-signed-certs\""
Sep 15 10:47:35.662120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.661900    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-44134e\"/\"default-dockercfg-dkv6h\""
Sep 15 10:47:35.669193 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.669170    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"]
Sep 15 10:47:35.761867 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.761828    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn9vm\" (UniqueName: \"kubernetes.io/projected/c337e105-cc31-4e0e-8b87-31bb66fe98d3-kube-api-access-tn9vm\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.761867 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.761871    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-model-cache\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.762081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.761902    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-dshm\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.762081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.761996    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tls-certs\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.762081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.762039    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-home\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.762081 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.762058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tmp-dir\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.862834 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.862777    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn9vm\" (UniqueName: \"kubernetes.io/projected/c337e105-cc31-4e0e-8b87-31bb66fe98d3-kube-api-access-tn9vm\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.863096 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.862853    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-model-cache\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.863096 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.862898    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-dshm\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.863096 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.862990    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tls-certs\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.863096 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.863038    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-home\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.863096 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.863063    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tmp-dir\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.863398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.863314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-model-cache\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.863465 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.863434    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tmp-dir\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.863550 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.863526    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-home\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.865531 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.865504    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-dshm\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.866471 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.866448    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tls-certs\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.870304 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.870281    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn9vm\" (UniqueName: \"kubernetes.io/projected/c337e105-cc31-4e0e-8b87-31bb66fe98d3-kube-api-access-tn9vm\") pod \"sub-test-model-44134e-kserve-6cd7886dcc-sktsx\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") " pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:35.968657 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:35.968555    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:36.247997 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:36.247970    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"]
Sep 15 10:47:36.250568 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:47:36.250543    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc337e105_cc31_4e0e_8b87_31bb66fe98d3.slice/crio-367d53936488d50b8bd81a408f22b845863427eea45fbfd34b8d1eefefc7b8b8 WatchSource:0}: Error finding container 367d53936488d50b8bd81a408f22b845863427eea45fbfd34b8d1eefefc7b8b8: Status 404 returned error can't find the container with id 367d53936488d50b8bd81a408f22b845863427eea45fbfd34b8d1eefefc7b8b8
Sep 15 10:47:37.203672 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:37.203629    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx" event={"ID":"c337e105-cc31-4e0e-8b87-31bb66fe98d3","Type":"ContainerStarted","Data":"a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9"}
Sep 15 10:47:37.204106 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:37.203680    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx" event={"ID":"c337e105-cc31-4e0e-8b87-31bb66fe98d3","Type":"ContainerStarted","Data":"367d53936488d50b8bd81a408f22b845863427eea45fbfd34b8d1eefefc7b8b8"}
Sep 15 10:47:37.220155 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:37.220107    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx" podStartSLOduration=2.220093348 podStartE2EDuration="2.220093348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:47:37.218821753 +0000 UTC m=+1775.912977431" watchObservedRunningTime="2026-09-15 10:47:37.220093348 +0000 UTC m=+1775.914249028"
Sep 15 10:47:39.457564 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:39.457526    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:47:45.969239 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:45.969197    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:45.969640 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:45.969255    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:45.981868 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:45.981843    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:47:46.255989 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:47:46.255960    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:48:14.351948 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:14.351907    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:48:29.458822 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:29.457062    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:48:38.503951 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:38.503907    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"]
Sep 15 10:48:39.483923 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:39.483890    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"]
Sep 15 10:48:42.848500 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:42.848459    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:48:52.748180 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:52.748144    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:48:53.524667 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.524608    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v" podUID="e629a6b1-352b-4b37-b56d-e2c104be6016" containerName="main" containerID="cri-o://f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5" gracePeriod=45
Sep 15 10:48:53.766211 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.766187    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:48:53.900986 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.900903    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e629a6b1-352b-4b37-b56d-e2c104be6016-tls-certs\") pod \"e629a6b1-352b-4b37-b56d-e2c104be6016\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") "
Sep 15 10:48:53.900986 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.900941    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-home\") pod \"e629a6b1-352b-4b37-b56d-e2c104be6016\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") "
Sep 15 10:48:53.900986 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.900960    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-dshm\") pod \"e629a6b1-352b-4b37-b56d-e2c104be6016\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") "
Sep 15 10:48:53.901241 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.901003    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-tmp-dir\") pod \"e629a6b1-352b-4b37-b56d-e2c104be6016\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") "
Sep 15 10:48:53.901241 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.901061    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-model-cache\") pod \"e629a6b1-352b-4b37-b56d-e2c104be6016\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") "
Sep 15 10:48:53.901241 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.901111    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mcr7\" (UniqueName: \"kubernetes.io/projected/e629a6b1-352b-4b37-b56d-e2c104be6016-kube-api-access-9mcr7\") pod \"e629a6b1-352b-4b37-b56d-e2c104be6016\" (UID: \"e629a6b1-352b-4b37-b56d-e2c104be6016\") "
Sep 15 10:48:53.901412 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.901264    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-home" (OuterVolumeSpecName: "home") pod "e629a6b1-352b-4b37-b56d-e2c104be6016" (UID: "e629a6b1-352b-4b37-b56d-e2c104be6016"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:48:53.901412 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.901281    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e629a6b1-352b-4b37-b56d-e2c104be6016" (UID: "e629a6b1-352b-4b37-b56d-e2c104be6016"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:48:53.901412 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.901293    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-model-cache" (OuterVolumeSpecName: "model-cache") pod "e629a6b1-352b-4b37-b56d-e2c104be6016" (UID: "e629a6b1-352b-4b37-b56d-e2c104be6016"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:48:53.901855 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.901730    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:53.901855 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.901756    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:53.901855 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.901771    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:53.903420 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.903397    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-dshm" (OuterVolumeSpecName: "dshm") pod "e629a6b1-352b-4b37-b56d-e2c104be6016" (UID: "e629a6b1-352b-4b37-b56d-e2c104be6016"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:48:53.903528 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.903418    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e629a6b1-352b-4b37-b56d-e2c104be6016-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e629a6b1-352b-4b37-b56d-e2c104be6016" (UID: "e629a6b1-352b-4b37-b56d-e2c104be6016"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:48:53.903528 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:53.903471    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e629a6b1-352b-4b37-b56d-e2c104be6016-kube-api-access-9mcr7" (OuterVolumeSpecName: "kube-api-access-9mcr7") pod "e629a6b1-352b-4b37-b56d-e2c104be6016" (UID: "e629a6b1-352b-4b37-b56d-e2c104be6016"). InnerVolumeSpecName "kube-api-access-9mcr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:48:54.002973 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.002936    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e629a6b1-352b-4b37-b56d-e2c104be6016-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:54.002973 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.002970    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e629a6b1-352b-4b37-b56d-e2c104be6016-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:54.002973 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.002980    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mcr7\" (UniqueName: \"kubernetes.io/projected/e629a6b1-352b-4b37-b56d-e2c104be6016-kube-api-access-9mcr7\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:54.503879 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.503836    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx" podUID="c337e105-cc31-4e0e-8b87-31bb66fe98d3" containerName="main" containerID="cri-o://a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9" gracePeriod=45
Sep 15 10:48:54.572687 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.572649    2584 generic.go:356] "Generic (PLEG): container finished" podID="e629a6b1-352b-4b37-b56d-e2c104be6016" containerID="f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5" exitCode=0
Sep 15 10:48:54.572919 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.572696    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"
Sep 15 10:48:54.572919 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.572718    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v" event={"ID":"e629a6b1-352b-4b37-b56d-e2c104be6016","Type":"ContainerDied","Data":"f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5"}
Sep 15 10:48:54.572919 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.572756    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v" event={"ID":"e629a6b1-352b-4b37-b56d-e2c104be6016","Type":"ContainerDied","Data":"180565fe2b512af0db73138b871566e8fed1d891995d7120a2720097ec823270"}
Sep 15 10:48:54.572919 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.572778    2584 scope.go:117] "RemoveContainer" containerID="f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5"
Sep 15 10:48:54.594448 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.594418    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"]
Sep 15 10:48:54.596908 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.596882    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-83e814/sub-test-model-83e814-kserve-65874dd4df-6wj9v"]
Sep 15 10:48:54.615390 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.615371    2584 scope.go:117] "RemoveContainer" containerID="f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5"
Sep 15 10:48:54.615720 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:48:54.615696    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5\": container with ID starting with f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5 not found: ID does not exist" containerID="f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5"
Sep 15 10:48:54.615784 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.615729    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5"} err="failed to get container status \"f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5\": rpc error: code = NotFound desc = could not find container \"f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5\": container with ID starting with f73b57d29807293dd602fb0f8324e7049e370f17b11423e400e05a5cf37877d5 not found: ID does not exist"
Sep 15 10:48:54.752550 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.752528    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:48:54.912595 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.912502    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-home\") pod \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") "
Sep 15 10:48:54.912595 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.912569    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tls-certs\") pod \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") "
Sep 15 10:48:54.912595 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.912592    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-model-cache\") pod \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") "
Sep 15 10:48:54.913178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.912607    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tmp-dir\") pod \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") "
Sep 15 10:48:54.913178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.912674    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-dshm\") pod \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") "
Sep 15 10:48:54.913178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.912706    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn9vm\" (UniqueName: \"kubernetes.io/projected/c337e105-cc31-4e0e-8b87-31bb66fe98d3-kube-api-access-tn9vm\") pod \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\" (UID: \"c337e105-cc31-4e0e-8b87-31bb66fe98d3\") "
Sep 15 10:48:54.913178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.912950    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c337e105-cc31-4e0e-8b87-31bb66fe98d3" (UID: "c337e105-cc31-4e0e-8b87-31bb66fe98d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:48:54.913178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.912962    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-home" (OuterVolumeSpecName: "home") pod "c337e105-cc31-4e0e-8b87-31bb66fe98d3" (UID: "c337e105-cc31-4e0e-8b87-31bb66fe98d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:48:54.913178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.912978    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "c337e105-cc31-4e0e-8b87-31bb66fe98d3" (UID: "c337e105-cc31-4e0e-8b87-31bb66fe98d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:48:54.913178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.913080    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:54.913178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.913095    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:54.913178 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.913104    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:54.915005 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.914980    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-dshm" (OuterVolumeSpecName: "dshm") pod "c337e105-cc31-4e0e-8b87-31bb66fe98d3" (UID: "c337e105-cc31-4e0e-8b87-31bb66fe98d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:48:54.915005 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.914987    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c337e105-cc31-4e0e-8b87-31bb66fe98d3" (UID: "c337e105-cc31-4e0e-8b87-31bb66fe98d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:48:54.915155 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:54.915034    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c337e105-cc31-4e0e-8b87-31bb66fe98d3-kube-api-access-tn9vm" (OuterVolumeSpecName: "kube-api-access-tn9vm") pod "c337e105-cc31-4e0e-8b87-31bb66fe98d3" (UID: "c337e105-cc31-4e0e-8b87-31bb66fe98d3"). InnerVolumeSpecName "kube-api-access-tn9vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:48:55.013962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.013928    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c337e105-cc31-4e0e-8b87-31bb66fe98d3-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:55.013962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.013958    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn9vm\" (UniqueName: \"kubernetes.io/projected/c337e105-cc31-4e0e-8b87-31bb66fe98d3-kube-api-access-tn9vm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:55.013962 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.013970    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c337e105-cc31-4e0e-8b87-31bb66fe98d3-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:48:55.579607 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.579572    2584 generic.go:356] "Generic (PLEG): container finished" podID="c337e105-cc31-4e0e-8b87-31bb66fe98d3" containerID="a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9" exitCode=0
Sep 15 10:48:55.579820 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.579633    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"
Sep 15 10:48:55.579820 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.579677    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx" event={"ID":"c337e105-cc31-4e0e-8b87-31bb66fe98d3","Type":"ContainerDied","Data":"a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9"}
Sep 15 10:48:55.579820 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.579720    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx" event={"ID":"c337e105-cc31-4e0e-8b87-31bb66fe98d3","Type":"ContainerDied","Data":"367d53936488d50b8bd81a408f22b845863427eea45fbfd34b8d1eefefc7b8b8"}
Sep 15 10:48:55.579820 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.579739    2584 scope.go:117] "RemoveContainer" containerID="a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9"
Sep 15 10:48:55.588286 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.588272    2584 scope.go:117] "RemoveContainer" containerID="a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9"
Sep 15 10:48:55.588505 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:48:55.588486    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9\": container with ID starting with a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9 not found: ID does not exist" containerID="a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9"
Sep 15 10:48:55.588572 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.588510    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9"} err="failed to get container status \"a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9\": rpc error: code = NotFound desc = could not find container \"a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9\": container with ID starting with a80b2c74da7ebc966de3cd5ed97214f7ead824a913eaeedc43c898b8b30612f9 not found: ID does not exist"
Sep 15 10:48:55.599782 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.599757    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"]
Sep 15 10:48:55.601344 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.601325    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/sub-test-model-44134e-kserve-6cd7886dcc-sktsx"]
Sep 15 10:48:55.784870 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.784826    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c337e105-cc31-4e0e-8b87-31bb66fe98d3" path="/var/lib/kubelet/pods/c337e105-cc31-4e0e-8b87-31bb66fe98d3/volumes"
Sep 15 10:48:55.785331 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:48:55.785309    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e629a6b1-352b-4b37-b56d-e2c104be6016" path="/var/lib/kubelet/pods/e629a6b1-352b-4b37-b56d-e2c104be6016/volumes"
Sep 15 10:49:16.256868 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.256777    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"]
Sep 15 10:49:16.257353 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.257219    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c337e105-cc31-4e0e-8b87-31bb66fe98d3" containerName="main"
Sep 15 10:49:16.257353 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.257234    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c337e105-cc31-4e0e-8b87-31bb66fe98d3" containerName="main"
Sep 15 10:49:16.257353 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.257256    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e629a6b1-352b-4b37-b56d-e2c104be6016" containerName="main"
Sep 15 10:49:16.257353 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.257265    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e629a6b1-352b-4b37-b56d-e2c104be6016" containerName="main"
Sep 15 10:49:16.257353 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.257318    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e629a6b1-352b-4b37-b56d-e2c104be6016" containerName="main"
Sep 15 10:49:16.257353 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.257326    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c337e105-cc31-4e0e-8b87-31bb66fe98d3" containerName="main"
Sep 15 10:49:16.260453 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.260436    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.262722 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.262681    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-44134e\"/\"kube-root-ca.crt\""
Sep 15 10:49:16.263538 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.263517    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-44134e\"/\"default-dockercfg-dkv6h\""
Sep 15 10:49:16.263860 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.263841    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-44134e\"/\"openshift-service-ca.crt\""
Sep 15 10:49:16.264008 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.263985    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-44134e\"/\"rate-test-model-44134e-kserve-self-signed-certs\""
Sep 15 10:49:16.267675 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.267614    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"]
Sep 15 10:49:16.306893 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.306855    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tls-certs\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.306893 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.306895    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-model-cache\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.307098 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.306927    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw5mr\" (UniqueName: \"kubernetes.io/projected/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-kube-api-access-rw5mr\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.307098 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.306970    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tmp-dir\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.307098 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.307016    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-home\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.307098 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.307031    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-dshm\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.407670 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.407627    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-home\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.407670 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.407668    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-dshm\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.407930 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.407695    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tls-certs\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.407930 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.407715    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-model-cache\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.407930 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.407749    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw5mr\" (UniqueName: \"kubernetes.io/projected/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-kube-api-access-rw5mr\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.407930 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.407780    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tmp-dir\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.408164 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.408147    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-home\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.408218 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.408193    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-model-cache\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.408256 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.408246    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tmp-dir\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.410044 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.410023    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-dshm\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.410227 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.410208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tls-certs\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.414943 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.414924    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw5mr\" (UniqueName: \"kubernetes.io/projected/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-kube-api-access-rw5mr\") pod \"rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") " pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.572474 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.572361    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:16.846166 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:16.845927    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"]
Sep 15 10:49:16.848777 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:49:16.848751    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod563ee031_ca4a_41a5_abdc_dfa8892c2bd2.slice/crio-99c660037d979ae96129e022bf7f08791ea593ec73c9f7397d1a9d13d85b03e8 WatchSource:0}: Error finding container 99c660037d979ae96129e022bf7f08791ea593ec73c9f7397d1a9d13d85b03e8: Status 404 returned error can't find the container with id 99c660037d979ae96129e022bf7f08791ea593ec73c9f7397d1a9d13d85b03e8
Sep 15 10:49:17.691247 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:17.691207    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv" event={"ID":"563ee031-ca4a-41a5-abdc-dfa8892c2bd2","Type":"ContainerStarted","Data":"83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e"}
Sep 15 10:49:17.691247 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:17.691252    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv" event={"ID":"563ee031-ca4a-41a5-abdc-dfa8892c2bd2","Type":"ContainerStarted","Data":"99c660037d979ae96129e022bf7f08791ea593ec73c9f7397d1a9d13d85b03e8"}
Sep 15 10:49:17.709427 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:17.709376    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv" podStartSLOduration=1.7093610620000002 podStartE2EDuration="1.709361062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:49:17.707438984 +0000 UTC m=+1876.401594664" watchObservedRunningTime="2026-09-15 10:49:17.709361062 +0000 UTC m=+1876.403516815"
Sep 15 10:49:19.753362 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:19.753326    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:49:26.573185 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:26.573095    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:26.573185 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:26.573158    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:26.585714 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:26.585686    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:26.744617 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:26.744589    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:49:37.760583 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:37.760546    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:49:52.155069 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:52.155034    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:49:59.364070 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:49:59.364028    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"]
Sep 15 10:50:02.755296 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:02.755258    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:50:14.383388 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.383338    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv" podUID="563ee031-ca4a-41a5-abdc-dfa8892c2bd2" containerName="main" containerID="cri-o://83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e" gracePeriod=45
Sep 15 10:50:14.626572 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.626543    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:50:14.742210 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742183    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw5mr\" (UniqueName: \"kubernetes.io/projected/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-kube-api-access-rw5mr\") pod \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") "
Sep 15 10:50:14.742398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742230    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-model-cache\") pod \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") "
Sep 15 10:50:14.742398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742257    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tmp-dir\") pod \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") "
Sep 15 10:50:14.742398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742328    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-home\") pod \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") "
Sep 15 10:50:14.742398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742365    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-dshm\") pod \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") "
Sep 15 10:50:14.742627 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742403    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tls-certs\") pod \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\" (UID: \"563ee031-ca4a-41a5-abdc-dfa8892c2bd2\") "
Sep 15 10:50:14.742627 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742460    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "563ee031-ca4a-41a5-abdc-dfa8892c2bd2" (UID: "563ee031-ca4a-41a5-abdc-dfa8892c2bd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:50:14.742627 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742541    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "563ee031-ca4a-41a5-abdc-dfa8892c2bd2" (UID: "563ee031-ca4a-41a5-abdc-dfa8892c2bd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:50:14.742627 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742582    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-home" (OuterVolumeSpecName: "home") pod "563ee031-ca4a-41a5-abdc-dfa8892c2bd2" (UID: "563ee031-ca4a-41a5-abdc-dfa8892c2bd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:50:14.742770 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742708    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:50:14.742770 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742726    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:50:14.742770 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.742740    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:50:14.744505 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.744471    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-dshm" (OuterVolumeSpecName: "dshm") pod "563ee031-ca4a-41a5-abdc-dfa8892c2bd2" (UID: "563ee031-ca4a-41a5-abdc-dfa8892c2bd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:50:14.744650 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.744521    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-kube-api-access-rw5mr" (OuterVolumeSpecName: "kube-api-access-rw5mr") pod "563ee031-ca4a-41a5-abdc-dfa8892c2bd2" (UID: "563ee031-ca4a-41a5-abdc-dfa8892c2bd2"). InnerVolumeSpecName "kube-api-access-rw5mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:50:14.744650 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.744523    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "563ee031-ca4a-41a5-abdc-dfa8892c2bd2" (UID: "563ee031-ca4a-41a5-abdc-dfa8892c2bd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:50:14.843953 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.843913    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:50:14.844124 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.843949    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:50:14.844124 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.843983    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw5mr\" (UniqueName: \"kubernetes.io/projected/563ee031-ca4a-41a5-abdc-dfa8892c2bd2-kube-api-access-rw5mr\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:50:14.968088 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.968048    2584 generic.go:356] "Generic (PLEG): container finished" podID="563ee031-ca4a-41a5-abdc-dfa8892c2bd2" containerID="83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e" exitCode=0
Sep 15 10:50:14.968266 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.968094    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"
Sep 15 10:50:14.968266 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.968112    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv" event={"ID":"563ee031-ca4a-41a5-abdc-dfa8892c2bd2","Type":"ContainerDied","Data":"83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e"}
Sep 15 10:50:14.968266 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.968151    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv" event={"ID":"563ee031-ca4a-41a5-abdc-dfa8892c2bd2","Type":"ContainerDied","Data":"99c660037d979ae96129e022bf7f08791ea593ec73c9f7397d1a9d13d85b03e8"}
Sep 15 10:50:14.968266 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.968176    2584 scope.go:117] "RemoveContainer" containerID="83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e"
Sep 15 10:50:14.977062 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.977044    2584 scope.go:117] "RemoveContainer" containerID="83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e"
Sep 15 10:50:14.977322 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:50:14.977299    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e\": container with ID starting with 83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e not found: ID does not exist" containerID="83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e"
Sep 15 10:50:14.977390 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.977336    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e"} err="failed to get container status \"83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e\": rpc error: code = NotFound desc = could not find container \"83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e\": container with ID starting with 83225fd3ce01c5e7bc67049f56a004a27549d2b8e27a246030741b5eaacbf67e not found: ID does not exist"
Sep 15 10:50:14.987889 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.987864    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"]
Sep 15 10:50:14.995075 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:14.995022    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-44134e/rate-test-model-44134e-kserve-5c77d7f6ff-4dfpv"]
Sep 15 10:50:15.785009 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:15.784973    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563ee031-ca4a-41a5-abdc-dfa8892c2bd2" path="/var/lib/kubelet/pods/563ee031-ca4a-41a5-abdc-dfa8892c2bd2/volumes"
Sep 15 10:50:17.152487 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:17.152451    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:50:34.535251 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.535215    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"]
Sep 15 10:50:34.536297 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.536271    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563ee031-ca4a-41a5-abdc-dfa8892c2bd2" containerName="main"
Sep 15 10:50:34.536398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.536300    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="563ee031-ca4a-41a5-abdc-dfa8892c2bd2" containerName="main"
Sep 15 10:50:34.536458 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.536402    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="563ee031-ca4a-41a5-abdc-dfa8892c2bd2" containerName="main"
Sep 15 10:50:34.541151 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.541121    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.543430 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.543411    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9f5ea7-serving-cert\""
Sep 15 10:50:34.549567 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.549541    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"]
Sep 15 10:50:34.630004 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.629968    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.630202 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.630037    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-metrics-certs\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.630202 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.630062    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tz24\" (UniqueName: \"kubernetes.io/projected/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-kube-api-access-5tz24\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.630202 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.630090    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.730606 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.730573    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-metrics-certs\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.730788 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.730617    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tz24\" (UniqueName: \"kubernetes.io/projected/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-kube-api-access-5tz24\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.730788 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.730748    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.730914 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.730854    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.731397 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.731373    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.733130 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.733101    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-metrics-certs\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.733310 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.733199    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.737973 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.737950    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tz24\" (UniqueName: \"kubernetes.io/projected/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-kube-api-access-5tz24\") pod \"maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:34.854202 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:34.854115    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:35.127988 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:35.127899    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"]
Sep 15 10:50:35.131084 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:50:35.131055    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6a1d8f_cb93_4bc8_8151_8751b22aa72c.slice/crio-886a936e4f4cba3f437c7fb7da733ee4fb4cf4a0d3af555e6fe822bb0a1c577a WatchSource:0}: Error finding container 886a936e4f4cba3f437c7fb7da733ee4fb4cf4a0d3af555e6fe822bb0a1c577a: Status 404 returned error can't find the container with id 886a936e4f4cba3f437c7fb7da733ee4fb4cf4a0d3af555e6fe822bb0a1c577a
Sep 15 10:50:35.132450 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:35.132429    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 10:50:36.065585 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:36.065549    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r" event={"ID":"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c","Type":"ContainerStarted","Data":"d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6"}
Sep 15 10:50:36.066100 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:36.065599    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r" event={"ID":"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c","Type":"ContainerStarted","Data":"886a936e4f4cba3f437c7fb7da733ee4fb4cf4a0d3af555e6fe822bb0a1c577a"}
Sep 15 10:50:36.066100 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:36.065636    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:50:36.086712 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:36.086660    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r" podStartSLOduration=1.72356926 podStartE2EDuration="2.086644951s" totalImagesPullingTime="363.075691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:50:36.084817164 +0000 UTC m=+1954.778972835" watchObservedRunningTime="2026-09-15 10:50:36.086644951 +0000 UTC m=+1954.780800632"
Sep 15 10:50:39.353860 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:39.353819    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:50:42.075169 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:50:42.075140    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:51:02.063456 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:02.063376    2584 scope.go:117] "RemoveContainer" containerID="bdd6285e305ee8f97525bda580421b3dd09998cc34575cf17957be119774029c"
Sep 15 10:51:14.361647 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.361609    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"]
Sep 15 10:51:14.365313 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.365292    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.368403 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.368381    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5ffc6a\"/\"default-dockercfg-zdf7z\""
Sep 15 10:51:14.368403 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.368381    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5ffc6a\"/\"ipp-route-5ffc6a-kserve-self-signed-certs\""
Sep 15 10:51:14.368578 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.368383    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5ffc6a\"/\"kube-root-ca.crt\""
Sep 15 10:51:14.368578 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.368390    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5ffc6a\"/\"openshift-service-ca.crt\""
Sep 15 10:51:14.375070 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.375046    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"]
Sep 15 10:51:14.398009 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.397975    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-model-cache\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.398009 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.398010    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slm6n\" (UniqueName: \"kubernetes.io/projected/ef1fe9ca-eca8-4b29-bf89-b15b20777958-kube-api-access-slm6n\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.398206 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.398082    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-dshm\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.398206 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.398108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tls-certs\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.398206 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.398139    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tmp-dir\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.398309 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.398229    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-home\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.498793 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.498739    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-home\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.499028 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.498830    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-model-cache\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.499028 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.498853    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slm6n\" (UniqueName: \"kubernetes.io/projected/ef1fe9ca-eca8-4b29-bf89-b15b20777958-kube-api-access-slm6n\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.499028 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.498903    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-dshm\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.499028 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.498931    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tls-certs\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.499028 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.498972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tmp-dir\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.499312 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.499181    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-home\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.499312 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.499250    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-model-cache\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.499393 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.499336    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tmp-dir\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.501283 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.501258    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-dshm\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.501493 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.501472    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tls-certs\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.508217 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.508189    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slm6n\" (UniqueName: \"kubernetes.io/projected/ef1fe9ca-eca8-4b29-bf89-b15b20777958-kube-api-access-slm6n\") pod \"ipp-route-5ffc6a-kserve-6858455f59-9gsbs\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") " pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.674954 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.674851    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:14.945934 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:14.945905    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"]
Sep 15 10:51:14.948434 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:51:14.948399    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef1fe9ca_eca8_4b29_bf89_b15b20777958.slice/crio-519573d752a3b91e3dae247f16b05eff9f8e1c8316ab12a15ef6c539b3978ff1 WatchSource:0}: Error finding container 519573d752a3b91e3dae247f16b05eff9f8e1c8316ab12a15ef6c539b3978ff1: Status 404 returned error can't find the container with id 519573d752a3b91e3dae247f16b05eff9f8e1c8316ab12a15ef6c539b3978ff1
Sep 15 10:51:15.241301 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:15.241211    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs" event={"ID":"ef1fe9ca-eca8-4b29-bf89-b15b20777958","Type":"ContainerStarted","Data":"8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab"}
Sep 15 10:51:15.241301 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:15.241262    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs" event={"ID":"ef1fe9ca-eca8-4b29-bf89-b15b20777958","Type":"ContainerStarted","Data":"519573d752a3b91e3dae247f16b05eff9f8e1c8316ab12a15ef6c539b3978ff1"}
Sep 15 10:51:15.256366 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:15.256319    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs" podStartSLOduration=1.256302748 podStartE2EDuration="1.256302748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:51:15.255884327 +0000 UTC m=+1993.950040008" watchObservedRunningTime="2026-09-15 10:51:15.256302748 +0000 UTC m=+1993.950458429"
Sep 15 10:51:18.257214 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:18.257176    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:51:24.675334 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:24.675293    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:24.675770 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:24.675353    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:24.688130 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:24.688096    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:25.299815 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:25.299771    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:51:37.255867 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:37.255825    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:51:51.223632 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.223597    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"]
Sep 15 10:51:51.227095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.227075    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:51:51.229863 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.229840    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-1b5e28-dockercfg-hc5qc\""
Sep 15 10:51:51.230195 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.230177    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-1b5e28\""
Sep 15 10:51:51.239881 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.239859    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"]
Sep 15 10:51:51.326989 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.326954    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41740be8-1fb3-418f-8794-7fca96eae28c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc\" (UID: \"41740be8-1fb3-418f-8794-7fca96eae28c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:51:51.327157 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.327015    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwb2\" (UniqueName: \"kubernetes.io/projected/41740be8-1fb3-418f-8794-7fca96eae28c-kube-api-access-qrwb2\") pod \"payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc\" (UID: \"41740be8-1fb3-418f-8794-7fca96eae28c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:51:51.428313 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.428276    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwb2\" (UniqueName: \"kubernetes.io/projected/41740be8-1fb3-418f-8794-7fca96eae28c-kube-api-access-qrwb2\") pod \"payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc\" (UID: \"41740be8-1fb3-418f-8794-7fca96eae28c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:51:51.428493 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.428368    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41740be8-1fb3-418f-8794-7fca96eae28c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc\" (UID: \"41740be8-1fb3-418f-8794-7fca96eae28c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:51:51.429074 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.429056    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41740be8-1fb3-418f-8794-7fca96eae28c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc\" (UID: \"41740be8-1fb3-418f-8794-7fca96eae28c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:51:51.436124 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.436102    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwb2\" (UniqueName: \"kubernetes.io/projected/41740be8-1fb3-418f-8794-7fca96eae28c-kube-api-access-qrwb2\") pod \"payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc\" (UID: \"41740be8-1fb3-418f-8794-7fca96eae28c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:51:51.537988 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.537948    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:51:51.809641 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:51.809607    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"]
Sep 15 10:51:51.811703 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:51:51.811661    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41740be8_1fb3_418f_8794_7fca96eae28c.slice/crio-af0447c93ddcf22812ffe738f341d2f90be5826fbacea6e44c1b643a802caf85 WatchSource:0}: Error finding container af0447c93ddcf22812ffe738f341d2f90be5826fbacea6e44c1b643a802caf85: Status 404 returned error can't find the container with id af0447c93ddcf22812ffe738f341d2f90be5826fbacea6e44c1b643a802caf85
Sep 15 10:51:52.413532 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:52.413496    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc" event={"ID":"41740be8-1fb3-418f-8794-7fca96eae28c","Type":"ContainerStarted","Data":"92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d"}
Sep 15 10:51:52.413975 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:52.413557    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc" event={"ID":"41740be8-1fb3-418f-8794-7fca96eae28c","Type":"ContainerStarted","Data":"af0447c93ddcf22812ffe738f341d2f90be5826fbacea6e44c1b643a802caf85"}
Sep 15 10:51:52.413975 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:52.413590    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:51:52.429019 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:52.428973    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc" podStartSLOduration=1.428958133 podStartE2EDuration="1.428958133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:51:52.42716649 +0000 UTC m=+2031.121322174" watchObservedRunningTime="2026-09-15 10:51:52.428958133 +0000 UTC m=+2031.123113812"
Sep 15 10:51:56.356238 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:51:56.356197    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:52:03.420276 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:03.420243    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:52:19.694521 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:19.694481    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"]
Sep 15 10:52:19.695306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:19.695268    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc" podUID="41740be8-1fb3-418f-8794-7fca96eae28c" containerName="payload-pre-processing" containerID="cri-o://92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d" gracePeriod=30
Sep 15 10:52:19.949434 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:19.949366    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:52:20.002435 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.002378    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrwb2\" (UniqueName: \"kubernetes.io/projected/41740be8-1fb3-418f-8794-7fca96eae28c-kube-api-access-qrwb2\") pod \"41740be8-1fb3-418f-8794-7fca96eae28c\" (UID: \"41740be8-1fb3-418f-8794-7fca96eae28c\") "
Sep 15 10:52:20.002637 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.002532    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41740be8-1fb3-418f-8794-7fca96eae28c-plugins-config-volume\") pod \"41740be8-1fb3-418f-8794-7fca96eae28c\" (UID: \"41740be8-1fb3-418f-8794-7fca96eae28c\") "
Sep 15 10:52:20.002956 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.002933    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41740be8-1fb3-418f-8794-7fca96eae28c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41740be8-1fb3-418f-8794-7fca96eae28c" (UID: "41740be8-1fb3-418f-8794-7fca96eae28c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:52:20.004642 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.004618    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41740be8-1fb3-418f-8794-7fca96eae28c-kube-api-access-qrwb2" (OuterVolumeSpecName: "kube-api-access-qrwb2") pod "41740be8-1fb3-418f-8794-7fca96eae28c" (UID: "41740be8-1fb3-418f-8794-7fca96eae28c"). InnerVolumeSpecName "kube-api-access-qrwb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:52:20.104037 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.103981    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrwb2\" (UniqueName: \"kubernetes.io/projected/41740be8-1fb3-418f-8794-7fca96eae28c-kube-api-access-qrwb2\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:52:20.104037 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.104027    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41740be8-1fb3-418f-8794-7fca96eae28c-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:52:20.559098 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.559056    2584 generic.go:356] "Generic (PLEG): container finished" podID="41740be8-1fb3-418f-8794-7fca96eae28c" containerID="92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d" exitCode=0
Sep 15 10:52:20.559306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.559122    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"
Sep 15 10:52:20.559306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.559122    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc" event={"ID":"41740be8-1fb3-418f-8794-7fca96eae28c","Type":"ContainerDied","Data":"92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d"}
Sep 15 10:52:20.559306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.559273    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc" event={"ID":"41740be8-1fb3-418f-8794-7fca96eae28c","Type":"ContainerDied","Data":"af0447c93ddcf22812ffe738f341d2f90be5826fbacea6e44c1b643a802caf85"}
Sep 15 10:52:20.559306 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.559297    2584 scope.go:117] "RemoveContainer" containerID="92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d"
Sep 15 10:52:20.567977 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.567955    2584 scope.go:117] "RemoveContainer" containerID="92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d"
Sep 15 10:52:20.568242 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:52:20.568222    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d\": container with ID starting with 92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d not found: ID does not exist" containerID="92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d"
Sep 15 10:52:20.568283 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.568252    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d"} err="failed to get container status \"92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d\": rpc error: code = NotFound desc = could not find container \"92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d\": container with ID starting with 92b91379c90b8c4a29a6593b8fd75b66f922b04cd7d86f4f96e40310c139668d not found: ID does not exist"
Sep 15 10:52:20.593465 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.593432    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"]
Sep 15 10:52:20.610767 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:20.610734    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1b5e28-55df7b58b-mc8hc"]
Sep 15 10:52:21.786618 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:21.786576    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41740be8-1fb3-418f-8794-7fca96eae28c" path="/var/lib/kubelet/pods/41740be8-1fb3-418f-8794-7fca96eae28c/volumes"
Sep 15 10:52:24.751727 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:24.751630    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:52:42.858926 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:42.858889    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:52:50.665163 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:50.665117    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"]
Sep 15 10:52:56.149362 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:52:56.149325    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:53:05.684495 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:05.684447    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs" podUID="ef1fe9ca-eca8-4b29-bf89-b15b20777958" containerName="main" containerID="cri-o://8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab" gracePeriod=45
Sep 15 10:53:05.926350 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:05.926326    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:53:06.013187 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013150    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slm6n\" (UniqueName: \"kubernetes.io/projected/ef1fe9ca-eca8-4b29-bf89-b15b20777958-kube-api-access-slm6n\") pod \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") "
Sep 15 10:53:06.013382 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013193    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tmp-dir\") pod \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") "
Sep 15 10:53:06.013382 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013273    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-model-cache\") pod \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") "
Sep 15 10:53:06.013382 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013302    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-home\") pod \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") "
Sep 15 10:53:06.013382 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013360    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tls-certs\") pod \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") "
Sep 15 10:53:06.013590 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013390    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-dshm\") pod \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\" (UID: \"ef1fe9ca-eca8-4b29-bf89-b15b20777958\") "
Sep 15 10:53:06.013590 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013504    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef1fe9ca-eca8-4b29-bf89-b15b20777958" (UID: "ef1fe9ca-eca8-4b29-bf89-b15b20777958"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:53:06.013590 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013517    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef1fe9ca-eca8-4b29-bf89-b15b20777958" (UID: "ef1fe9ca-eca8-4b29-bf89-b15b20777958"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:53:06.013712 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013591    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-home" (OuterVolumeSpecName: "home") pod "ef1fe9ca-eca8-4b29-bf89-b15b20777958" (UID: "ef1fe9ca-eca8-4b29-bf89-b15b20777958"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:53:06.013750 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013708    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:53:06.013750 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013722    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:53:06.013750 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.013732    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:53:06.015497 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.015472    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-dshm" (OuterVolumeSpecName: "dshm") pod "ef1fe9ca-eca8-4b29-bf89-b15b20777958" (UID: "ef1fe9ca-eca8-4b29-bf89-b15b20777958"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:53:06.015591 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.015489    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef1fe9ca-eca8-4b29-bf89-b15b20777958" (UID: "ef1fe9ca-eca8-4b29-bf89-b15b20777958"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:53:06.015591 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.015523    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1fe9ca-eca8-4b29-bf89-b15b20777958-kube-api-access-slm6n" (OuterVolumeSpecName: "kube-api-access-slm6n") pod "ef1fe9ca-eca8-4b29-bf89-b15b20777958" (UID: "ef1fe9ca-eca8-4b29-bf89-b15b20777958"). InnerVolumeSpecName "kube-api-access-slm6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:53:06.114227 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.114175    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1fe9ca-eca8-4b29-bf89-b15b20777958-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:53:06.114227 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.114221    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1fe9ca-eca8-4b29-bf89-b15b20777958-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:53:06.114227 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.114232    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slm6n\" (UniqueName: \"kubernetes.io/projected/ef1fe9ca-eca8-4b29-bf89-b15b20777958-kube-api-access-slm6n\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:53:06.780100 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.780068    2584 generic.go:356] "Generic (PLEG): container finished" podID="ef1fe9ca-eca8-4b29-bf89-b15b20777958" containerID="8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab" exitCode=0
Sep 15 10:53:06.780594 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.780110    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"
Sep 15 10:53:06.780594 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.780151    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs" event={"ID":"ef1fe9ca-eca8-4b29-bf89-b15b20777958","Type":"ContainerDied","Data":"8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab"}
Sep 15 10:53:06.780594 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.780198    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs" event={"ID":"ef1fe9ca-eca8-4b29-bf89-b15b20777958","Type":"ContainerDied","Data":"519573d752a3b91e3dae247f16b05eff9f8e1c8316ab12a15ef6c539b3978ff1"}
Sep 15 10:53:06.780594 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.780222    2584 scope.go:117] "RemoveContainer" containerID="8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab"
Sep 15 10:53:06.791320 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.791300    2584 scope.go:117] "RemoveContainer" containerID="8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab"
Sep 15 10:53:06.791679 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:53:06.791650    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab\": container with ID starting with 8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab not found: ID does not exist" containerID="8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab"
Sep 15 10:53:06.791757 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.791691    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab"} err="failed to get container status \"8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab\": rpc error: code = NotFound desc = could not find container \"8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab\": container with ID starting with 8ef6ad975cdb2590fe8c6729334146c5230f5c5ad1c99cd6b53e1b62e87456ab not found: ID does not exist"
Sep 15 10:53:06.802837 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.802812    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"]
Sep 15 10:53:06.807036 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:06.807012    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-5ffc6a/ipp-route-5ffc6a-kserve-6858455f59-9gsbs"]
Sep 15 10:53:07.786854 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:07.786793    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef1fe9ca-eca8-4b29-bf89-b15b20777958" path="/var/lib/kubelet/pods/ef1fe9ca-eca8-4b29-bf89-b15b20777958/volumes"
Sep 15 10:53:44.212323 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.212285    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"]
Sep 15 10:53:44.212912 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.212531    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r" podUID="5f6a1d8f-cb93-4bc8-8151-8751b22aa72c" containerName="maas-api" containerID="cri-o://d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6" gracePeriod=30
Sep 15 10:53:44.453535 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.453511    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:53:44.560378 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.560339    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-metrics-certs\") pod \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") "
Sep 15 10:53:44.560378 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.560394    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-maas-api-tls\") pod \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") "
Sep 15 10:53:44.560618 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.560417    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-openshift-service-ca\") pod \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") "
Sep 15 10:53:44.560618 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.560458    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tz24\" (UniqueName: \"kubernetes.io/projected/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-kube-api-access-5tz24\") pod \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\" (UID: \"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c\") "
Sep 15 10:53:44.560868 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.560840    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f6a1d8f-cb93-4bc8-8151-8751b22aa72c" (UID: "5f6a1d8f-cb93-4bc8-8151-8751b22aa72c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:53:44.562688 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.562662    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-kube-api-access-5tz24" (OuterVolumeSpecName: "kube-api-access-5tz24") pod "5f6a1d8f-cb93-4bc8-8151-8751b22aa72c" (UID: "5f6a1d8f-cb93-4bc8-8151-8751b22aa72c"). InnerVolumeSpecName "kube-api-access-5tz24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:53:44.562688 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.562671    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f6a1d8f-cb93-4bc8-8151-8751b22aa72c" (UID: "5f6a1d8f-cb93-4bc8-8151-8751b22aa72c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:53:44.562822 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.562744    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5f6a1d8f-cb93-4bc8-8151-8751b22aa72c" (UID: "5f6a1d8f-cb93-4bc8-8151-8751b22aa72c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:53:44.661361 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.661326    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-metrics-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:53:44.661361 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.661354    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-maas-api-tls\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:53:44.661361 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.661363    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-openshift-service-ca\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:53:44.661599 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.661374    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tz24\" (UniqueName: \"kubernetes.io/projected/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c-kube-api-access-5tz24\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:53:44.951792 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.951703    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f6a1d8f-cb93-4bc8-8151-8751b22aa72c" containerID="d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6" exitCode=0
Sep 15 10:53:44.951792 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.951744    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"
Sep 15 10:53:44.951792 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.951791    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r" event={"ID":"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c","Type":"ContainerDied","Data":"d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6"}
Sep 15 10:53:44.952035 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.951863    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r" event={"ID":"5f6a1d8f-cb93-4bc8-8151-8751b22aa72c","Type":"ContainerDied","Data":"886a936e4f4cba3f437c7fb7da733ee4fb4cf4a0d3af555e6fe822bb0a1c577a"}
Sep 15 10:53:44.952035 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.951888    2584 scope.go:117] "RemoveContainer" containerID="d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6"
Sep 15 10:53:44.961164 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.961146    2584 scope.go:117] "RemoveContainer" containerID="d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6"
Sep 15 10:53:44.961415 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:53:44.961397    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6\": container with ID starting with d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6 not found: ID does not exist" containerID="d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6"
Sep 15 10:53:44.961481 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.961428    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6"} err="failed to get container status \"d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6\": rpc error: code = NotFound desc = could not find container \"d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6\": container with ID starting with d02c457f1be9e7320e61483e4fb7c0d2b49e7660bb456be30f70cbfe0b9d7ea6 not found: ID does not exist"
Sep 15 10:53:44.971685 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.971652    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"]
Sep 15 10:53:44.979703 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:44.979681    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5ea7-68868c65f8-jdv5r"]
Sep 15 10:53:45.785370 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:45.785336    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f6a1d8f-cb93-4bc8-8151-8751b22aa72c" path="/var/lib/kubelet/pods/5f6a1d8f-cb93-4bc8-8151-8751b22aa72c/volumes"
Sep 15 10:53:47.748793 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:53:47.748755    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:54:01.700703 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:01.700620    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c"]
Sep 15 10:54:01.705295 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:01.705262    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-83e814-29824485-zmt4c"]
Sep 15 10:54:01.731196 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:01.731153    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"]
Sep 15 10:54:01.731463 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:01.731418    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk" podUID="78ab1f0a-494e-4c75-a991-03b236ceb215" containerName="payload-processing" containerID="cri-o://2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf" gracePeriod=30
Sep 15 10:54:01.787281 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:01.787248    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de7d7fa2-9207-47c7-bb26-9c1fa07ef130" path="/var/lib/kubelet/pods/de7d7fa2-9207-47c7-bb26-9c1fa07ef130/volumes"
Sep 15 10:54:01.995279 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:01.995251    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:54:02.039488 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.039452    2584 generic.go:356] "Generic (PLEG): container finished" podID="78ab1f0a-494e-4c75-a991-03b236ceb215" containerID="2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf" exitCode=0
Sep 15 10:54:02.039655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.039502    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"
Sep 15 10:54:02.039655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.039528    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk" event={"ID":"78ab1f0a-494e-4c75-a991-03b236ceb215","Type":"ContainerDied","Data":"2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf"}
Sep 15 10:54:02.039655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.039580    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk" event={"ID":"78ab1f0a-494e-4c75-a991-03b236ceb215","Type":"ContainerDied","Data":"a53eaf2ca0b2c1e64cb906f0cfc177a7b2a89afac2374e1f7430a42805650fca"}
Sep 15 10:54:02.039655 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.039605    2584 scope.go:117] "RemoveContainer" containerID="2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf"
Sep 15 10:54:02.048282 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.048261    2584 scope.go:117] "RemoveContainer" containerID="2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf"
Sep 15 10:54:02.048589 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:54:02.048560    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf\": container with ID starting with 2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf not found: ID does not exist" containerID="2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf"
Sep 15 10:54:02.048669 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.048597    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf"} err="failed to get container status \"2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf\": rpc error: code = NotFound desc = could not find container \"2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf\": container with ID starting with 2e13e03263efb93a39c24e61c27012ab335dce3f6b8edf657083e96f55880bdf not found: ID does not exist"
Sep 15 10:54:02.102387 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.102353    2584 scope.go:117] "RemoveContainer" containerID="2a4fc88e7ae29c4b6cfe25d3064b4def716ef93ceb7e0db60a04e4ee4f8f5b2c"
Sep 15 10:54:02.118770 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.118749    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ab1f0a-494e-4c75-a991-03b236ceb215-plugins-config-volume\") pod \"78ab1f0a-494e-4c75-a991-03b236ceb215\" (UID: \"78ab1f0a-494e-4c75-a991-03b236ceb215\") "
Sep 15 10:54:02.118866 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.118785    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcgsz\" (UniqueName: \"kubernetes.io/projected/78ab1f0a-494e-4c75-a991-03b236ceb215-kube-api-access-wcgsz\") pod \"78ab1f0a-494e-4c75-a991-03b236ceb215\" (UID: \"78ab1f0a-494e-4c75-a991-03b236ceb215\") "
Sep 15 10:54:02.119132 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.119110    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ab1f0a-494e-4c75-a991-03b236ceb215-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78ab1f0a-494e-4c75-a991-03b236ceb215" (UID: "78ab1f0a-494e-4c75-a991-03b236ceb215"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:54:02.121014 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.120992    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ab1f0a-494e-4c75-a991-03b236ceb215-kube-api-access-wcgsz" (OuterVolumeSpecName: "kube-api-access-wcgsz") pod "78ab1f0a-494e-4c75-a991-03b236ceb215" (UID: "78ab1f0a-494e-4c75-a991-03b236ceb215"). InnerVolumeSpecName "kube-api-access-wcgsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:54:02.220176 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.220139    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ab1f0a-494e-4c75-a991-03b236ceb215-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:54:02.220176 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.220177    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcgsz\" (UniqueName: \"kubernetes.io/projected/78ab1f0a-494e-4c75-a991-03b236ceb215-kube-api-access-wcgsz\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:54:02.357914 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.357876    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"]
Sep 15 10:54:02.361305 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:02.361274    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-83e814-7bddcc98c7-mjhkk"]
Sep 15 10:54:03.784839 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:03.784785    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78ab1f0a-494e-4c75-a991-03b236ceb215" path="/var/lib/kubelet/pods/78ab1f0a-494e-4c75-a991-03b236ceb215/volumes"
Sep 15 10:54:04.648743 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:04.648706    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:54:19.412212 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.412172    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"]
Sep 15 10:54:19.412885 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.412463    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644" podUID="56d52716-58b7-4e1e-b1e6-6fe03046e008" containerName="maas-api" containerID="cri-o://b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343" gracePeriod=30
Sep 15 10:54:19.467965 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.467931    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"]
Sep 15 10:54:19.468209 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.468180    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht" podUID="2ec04348-d8fb-449a-ba54-67d04449a134" containerName="payload-processing" containerID="cri-o://b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd" gracePeriod=30
Sep 15 10:54:19.712686 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.712655    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:54:19.743791 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.743762    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:54:19.787612 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.787588    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz92h\" (UniqueName: \"kubernetes.io/projected/2ec04348-d8fb-449a-ba54-67d04449a134-kube-api-access-nz92h\") pod \"2ec04348-d8fb-449a-ba54-67d04449a134\" (UID: \"2ec04348-d8fb-449a-ba54-67d04449a134\") "
Sep 15 10:54:19.787779 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.787624    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-maas-api-tls\") pod \"56d52716-58b7-4e1e-b1e6-6fe03046e008\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") "
Sep 15 10:54:19.787779 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.787674    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d52716-58b7-4e1e-b1e6-6fe03046e008-openshift-service-ca\") pod \"56d52716-58b7-4e1e-b1e6-6fe03046e008\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") "
Sep 15 10:54:19.787779 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.787715    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms2hq\" (UniqueName: \"kubernetes.io/projected/56d52716-58b7-4e1e-b1e6-6fe03046e008-kube-api-access-ms2hq\") pod \"56d52716-58b7-4e1e-b1e6-6fe03046e008\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") "
Sep 15 10:54:19.787779 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.787737    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec04348-d8fb-449a-ba54-67d04449a134-plugins-config-volume\") pod \"2ec04348-d8fb-449a-ba54-67d04449a134\" (UID: \"2ec04348-d8fb-449a-ba54-67d04449a134\") "
Sep 15 10:54:19.787779 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.787760    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-metrics-certs\") pod \"56d52716-58b7-4e1e-b1e6-6fe03046e008\" (UID: \"56d52716-58b7-4e1e-b1e6-6fe03046e008\") "
Sep 15 10:54:19.788134 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.788104    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d52716-58b7-4e1e-b1e6-6fe03046e008-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56d52716-58b7-4e1e-b1e6-6fe03046e008" (UID: "56d52716-58b7-4e1e-b1e6-6fe03046e008"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:54:19.788199 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.788155    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec04348-d8fb-449a-ba54-67d04449a134-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ec04348-d8fb-449a-ba54-67d04449a134" (UID: "2ec04348-d8fb-449a-ba54-67d04449a134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:54:19.789768 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.789746    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec04348-d8fb-449a-ba54-67d04449a134-kube-api-access-nz92h" (OuterVolumeSpecName: "kube-api-access-nz92h") pod "2ec04348-d8fb-449a-ba54-67d04449a134" (UID: "2ec04348-d8fb-449a-ba54-67d04449a134"). InnerVolumeSpecName "kube-api-access-nz92h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:54:19.789899 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.789772    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "56d52716-58b7-4e1e-b1e6-6fe03046e008" (UID: "56d52716-58b7-4e1e-b1e6-6fe03046e008"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:54:19.790111 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.790090    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56d52716-58b7-4e1e-b1e6-6fe03046e008" (UID: "56d52716-58b7-4e1e-b1e6-6fe03046e008"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:54:19.790208 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.790167    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d52716-58b7-4e1e-b1e6-6fe03046e008-kube-api-access-ms2hq" (OuterVolumeSpecName: "kube-api-access-ms2hq") pod "56d52716-58b7-4e1e-b1e6-6fe03046e008" (UID: "56d52716-58b7-4e1e-b1e6-6fe03046e008"). InnerVolumeSpecName "kube-api-access-ms2hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:54:19.888947 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.888910    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz92h\" (UniqueName: \"kubernetes.io/projected/2ec04348-d8fb-449a-ba54-67d04449a134-kube-api-access-nz92h\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:54:19.888947 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.888940    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-maas-api-tls\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:54:19.888947 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.888950    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d52716-58b7-4e1e-b1e6-6fe03046e008-openshift-service-ca\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:54:19.888947 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.888959    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms2hq\" (UniqueName: \"kubernetes.io/projected/56d52716-58b7-4e1e-b1e6-6fe03046e008-kube-api-access-ms2hq\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:54:19.889239 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.888967    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec04348-d8fb-449a-ba54-67d04449a134-plugins-config-volume\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:54:19.889239 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:19.888976    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d52716-58b7-4e1e-b1e6-6fe03046e008-metrics-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:54:20.121574 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.121540    2584 generic.go:356] "Generic (PLEG): container finished" podID="56d52716-58b7-4e1e-b1e6-6fe03046e008" containerID="b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343" exitCode=0
Sep 15 10:54:20.121782 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.121587    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"
Sep 15 10:54:20.121782 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.121584    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644" event={"ID":"56d52716-58b7-4e1e-b1e6-6fe03046e008","Type":"ContainerDied","Data":"b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343"}
Sep 15 10:54:20.121782 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.121716    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644" event={"ID":"56d52716-58b7-4e1e-b1e6-6fe03046e008","Type":"ContainerDied","Data":"10a088dd3828cd0b3c20ae186a670d4c08ccfc0113773a36753dada41436931d"}
Sep 15 10:54:20.121782 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.121749    2584 scope.go:117] "RemoveContainer" containerID="b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343"
Sep 15 10:54:20.123042 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.122979    2584 generic.go:356] "Generic (PLEG): container finished" podID="2ec04348-d8fb-449a-ba54-67d04449a134" containerID="b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd" exitCode=0
Sep 15 10:54:20.123042 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.123016    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"
Sep 15 10:54:20.123042 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.123031    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht" event={"ID":"2ec04348-d8fb-449a-ba54-67d04449a134","Type":"ContainerDied","Data":"b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd"}
Sep 15 10:54:20.123171 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.123072    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht" event={"ID":"2ec04348-d8fb-449a-ba54-67d04449a134","Type":"ContainerDied","Data":"fa90a7df0fe2b3d9ec21bbc700a2fd2a05b053b5fad8caf7b2d787989e46e69d"}
Sep 15 10:54:20.130516 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.130497    2584 scope.go:117] "RemoveContainer" containerID="b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343"
Sep 15 10:54:20.130896 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:54:20.130869    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343\": container with ID starting with b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343 not found: ID does not exist" containerID="b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343"
Sep 15 10:54:20.130978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.130907    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343"} err="failed to get container status \"b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343\": rpc error: code = NotFound desc = could not find container \"b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343\": container with ID starting with b3d95bb3d22a831065e896748d4e25c4fd3369bd1eeca5e7dd8e45c64e614343 not found: ID does not exist"
Sep 15 10:54:20.130978 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.130930    2584 scope.go:117] "RemoveContainer" containerID="b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd"
Sep 15 10:54:20.138662 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.138643    2584 scope.go:117] "RemoveContainer" containerID="b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd"
Sep 15 10:54:20.138954 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:54:20.138936    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd\": container with ID starting with b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd not found: ID does not exist" containerID="b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd"
Sep 15 10:54:20.139008 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.138961    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd"} err="failed to get container status \"b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd\": rpc error: code = NotFound desc = could not find container \"b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd\": container with ID starting with b2558d0aa9bb2938ef4ef54dfaca0f059479339850965253ffbf4c2e1aa6befd not found: ID does not exist"
Sep 15 10:54:20.148850 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.148825    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"]
Sep 15 10:54:20.152738 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.152716    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-44134e-779d9b64cf-8qnht"]
Sep 15 10:54:20.162399 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.162372    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"]
Sep 15 10:54:20.163755 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:20.163737    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-44134e-574787566c-xv644"]
Sep 15 10:54:21.784280 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:21.784249    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ec04348-d8fb-449a-ba54-67d04449a134" path="/var/lib/kubelet/pods/2ec04348-d8fb-449a-ba54-67d04449a134/volumes"
Sep 15 10:54:21.784671 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:21.784632    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56d52716-58b7-4e1e-b1e6-6fe03046e008" path="/var/lib/kubelet/pods/56d52716-58b7-4e1e-b1e6-6fe03046e008/volumes"
Sep 15 10:54:22.249883 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:22.249849    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:54:30.063864 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:54:30.063823    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:55:53.458238 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:55:53.458197    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:56:19.454333 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:56:19.454290    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:56:27.450161 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:56:27.450121    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:56:34.651001 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:56:34.650963    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:56:41.658622 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:56:41.658585    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:56:56.368184 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:56:56.368106    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:57:04.867622 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:57:04.867584    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:58:32.552524 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:32.552439    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:58:40.555433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:40.555397    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:58:49.058479 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:49.058439    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:58:51.954671 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.954633    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"]
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.954987    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78ab1f0a-494e-4c75-a991-03b236ceb215" containerName="payload-processing"
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955000    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="78ab1f0a-494e-4c75-a991-03b236ceb215" containerName="payload-processing"
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955011    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41740be8-1fb3-418f-8794-7fca96eae28c" containerName="payload-pre-processing"
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955017    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="41740be8-1fb3-418f-8794-7fca96eae28c" containerName="payload-pre-processing"
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955026    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec04348-d8fb-449a-ba54-67d04449a134" containerName="payload-processing"
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955032    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec04348-d8fb-449a-ba54-67d04449a134" containerName="payload-processing"
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955039    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d52716-58b7-4e1e-b1e6-6fe03046e008" containerName="maas-api"
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955044    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d52716-58b7-4e1e-b1e6-6fe03046e008" containerName="maas-api"
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955051    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef1fe9ca-eca8-4b29-bf89-b15b20777958" containerName="main"
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955056    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef1fe9ca-eca8-4b29-bf89-b15b20777958" containerName="main"
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955064    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f6a1d8f-cb93-4bc8-8151-8751b22aa72c" containerName="maas-api"
Sep 15 10:58:51.955095 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955070    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f6a1d8f-cb93-4bc8-8151-8751b22aa72c" containerName="maas-api"
Sep 15 10:58:51.955637 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955122    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ec04348-d8fb-449a-ba54-67d04449a134" containerName="payload-processing"
Sep 15 10:58:51.955637 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955130    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="41740be8-1fb3-418f-8794-7fca96eae28c" containerName="payload-pre-processing"
Sep 15 10:58:51.955637 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955137    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef1fe9ca-eca8-4b29-bf89-b15b20777958" containerName="main"
Sep 15 10:58:51.955637 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955143    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d52716-58b7-4e1e-b1e6-6fe03046e008" containerName="maas-api"
Sep 15 10:58:51.955637 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955150    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f6a1d8f-cb93-4bc8-8151-8751b22aa72c" containerName="maas-api"
Sep 15 10:58:51.955637 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.955155    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="78ab1f0a-494e-4c75-a991-03b236ceb215" containerName="payload-processing"
Sep 15 10:58:51.958428 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.958410    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:51.960575 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.960552    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-eb5ed9e6-kserve-self-signed-certs\""
Sep 15 10:58:51.965052 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:51.965012    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"]
Sep 15 10:58:52.086818 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.086765    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-home\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.086818 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.086822    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e9fd4e-111a-44c2-9ff2-6c2347444092-tls-certs\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.087047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.086911    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-model-cache\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.087047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.086962    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-dshm\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.087047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.087008    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-tmp-dir\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.087047 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.087025    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp2kh\" (UniqueName: \"kubernetes.io/projected/11e9fd4e-111a-44c2-9ff2-6c2347444092-kube-api-access-lp2kh\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.188365 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.188330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-home\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.188365 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.188367    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e9fd4e-111a-44c2-9ff2-6c2347444092-tls-certs\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.188623 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.188400    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-model-cache\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.188623 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.188437    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-dshm\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.188623 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.188464    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-tmp-dir\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.188623 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.188479    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp2kh\" (UniqueName: \"kubernetes.io/projected/11e9fd4e-111a-44c2-9ff2-6c2347444092-kube-api-access-lp2kh\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.188882 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.188829    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-home\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.188953 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.188878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-tmp-dir\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.188953 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.188934    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-model-cache\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.190633 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.190615    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-dshm\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.190919 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.190901    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e9fd4e-111a-44c2-9ff2-6c2347444092-tls-certs\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.195624 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.195602    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp2kh\" (UniqueName: \"kubernetes.io/projected/11e9fd4e-111a-44c2-9ff2-6c2347444092-kube-api-access-lp2kh\") pod \"e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") " pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.270193 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.270163    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:58:52.543171 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:52.543093    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"]
Sep 15 10:58:52.546021 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:58:52.545991    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11e9fd4e_111a_44c2_9ff2_6c2347444092.slice/crio-a112cd704d19c915fa343573de67bcd0a955b9c5aab29d57e7b963fb145f6d6f WatchSource:0}: Error finding container a112cd704d19c915fa343573de67bcd0a955b9c5aab29d57e7b963fb145f6d6f: Status 404 returned error can't find the container with id a112cd704d19c915fa343573de67bcd0a955b9c5aab29d57e7b963fb145f6d6f
Sep 15 10:58:53.320575 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:53.320537    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg" event={"ID":"11e9fd4e-111a-44c2-9ff2-6c2347444092","Type":"ContainerStarted","Data":"b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29"}
Sep 15 10:58:53.321066 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:53.320587    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg" event={"ID":"11e9fd4e-111a-44c2-9ff2-6c2347444092","Type":"ContainerStarted","Data":"a112cd704d19c915fa343573de67bcd0a955b9c5aab29d57e7b963fb145f6d6f"}
Sep 15 10:58:53.337620 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:53.337567    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg" podStartSLOduration=2.337551398 podStartE2EDuration="2.337551398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:58:53.335862765 +0000 UTC m=+2452.030018443" watchObservedRunningTime="2026-09-15 10:58:53.337551398 +0000 UTC m=+2452.031707078"
Sep 15 10:58:55.756709 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:55.756671    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:58:59.249575 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.249538    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"]
Sep 15 10:58:59.253458 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.253427    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.255791 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.255763    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-eb5ed9e6-kserve-self-signed-certs\""
Sep 15 10:58:59.260602 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.260577    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"]
Sep 15 10:58:59.350794 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.350757    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-dshm\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.350794 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.350790    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-home\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.351049 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.350843    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a256285e-2296-4c70-ba7c-3e4cdb479b63-tls-certs\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.351049 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.350863    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-model-cache\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.351049 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.350894    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-tmp-dir\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.351049 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.350973    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlsw7\" (UniqueName: \"kubernetes.io/projected/a256285e-2296-4c70-ba7c-3e4cdb479b63-kube-api-access-qlsw7\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.451928 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.451894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-model-cache\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.452113 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.451948    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-tmp-dir\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.452113 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.452034    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlsw7\" (UniqueName: \"kubernetes.io/projected/a256285e-2296-4c70-ba7c-3e4cdb479b63-kube-api-access-qlsw7\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.452113 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.452085    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-dshm\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.452113 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.452102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-home\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.452317 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.452134    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a256285e-2296-4c70-ba7c-3e4cdb479b63-tls-certs\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.452317 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.452280    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-model-cache\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.452436 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.452413    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-tmp-dir\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.452491 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.452473    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-home\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.454369 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.454349    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-dshm\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.454467 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.454450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a256285e-2296-4c70-ba7c-3e4cdb479b63-tls-certs\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.459723 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.459702    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlsw7\" (UniqueName: \"kubernetes.io/projected/a256285e-2296-4c70-ba7c-3e4cdb479b63-kube-api-access-qlsw7\") pod \"e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") " pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.564257 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.564176    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:58:59.836586 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:58:59.836508    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"]
Sep 15 10:58:59.839764 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:58:59.839739    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda256285e_2296_4c70_ba7c_3e4cdb479b63.slice/crio-bd210f8b541202b283ef5b08ab27dfd58e04b7eaaa02782a58158458e5840dca WatchSource:0}: Error finding container bd210f8b541202b283ef5b08ab27dfd58e04b7eaaa02782a58158458e5840dca: Status 404 returned error can't find the container with id bd210f8b541202b283ef5b08ab27dfd58e04b7eaaa02782a58158458e5840dca
Sep 15 10:59:00.352035 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:00.351997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr" event={"ID":"a256285e-2296-4c70-ba7c-3e4cdb479b63","Type":"ContainerStarted","Data":"3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8"}
Sep 15 10:59:00.352035 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:00.352040    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr" event={"ID":"a256285e-2296-4c70-ba7c-3e4cdb479b63","Type":"ContainerStarted","Data":"bd210f8b541202b283ef5b08ab27dfd58e04b7eaaa02782a58158458e5840dca"}
Sep 15 10:59:00.368785 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:00.368735    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr" podStartSLOduration=1.368720114 podStartE2EDuration="1.368720114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:59:00.366796304 +0000 UTC m=+2459.060951986" watchObservedRunningTime="2026-09-15 10:59:00.368720114 +0000 UTC m=+2459.062875793"
Sep 15 10:59:02.270734 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:02.270684    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:59:02.271165 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:02.270752    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:59:02.283288 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:02.283259    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:59:02.373045 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:02.373014    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:59:08.078003 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:08.077964    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:59:09.564645 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:09.564615    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:59:09.564645 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:09.564656    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:59:09.577156 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:09.577132    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:59:10.406349 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:10.406319    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:59:24.979185 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:24.979148    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"]
Sep 15 10:59:26.741173 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:26.741133    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"]
Sep 15 10:59:27.250107 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:27.250071    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:59:34.851533 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:34.851499    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:59:39.998289 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:39.998244    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg" podUID="11e9fd4e-111a-44c2-9ff2-6c2347444092" containerName="main" containerID="cri-o://b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29" gracePeriod=45
Sep 15 10:59:40.241754 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.241728    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:59:40.340585 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.340552    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-model-cache\") pod \"11e9fd4e-111a-44c2-9ff2-6c2347444092\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") "
Sep 15 10:59:40.340767 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.340613    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e9fd4e-111a-44c2-9ff2-6c2347444092-tls-certs\") pod \"11e9fd4e-111a-44c2-9ff2-6c2347444092\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") "
Sep 15 10:59:40.340767 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.340644    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-home\") pod \"11e9fd4e-111a-44c2-9ff2-6c2347444092\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") "
Sep 15 10:59:40.340767 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.340663    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp2kh\" (UniqueName: \"kubernetes.io/projected/11e9fd4e-111a-44c2-9ff2-6c2347444092-kube-api-access-lp2kh\") pod \"11e9fd4e-111a-44c2-9ff2-6c2347444092\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") "
Sep 15 10:59:40.340767 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.340691    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-dshm\") pod \"11e9fd4e-111a-44c2-9ff2-6c2347444092\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") "
Sep 15 10:59:40.340767 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.340716    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-tmp-dir\") pod \"11e9fd4e-111a-44c2-9ff2-6c2347444092\" (UID: \"11e9fd4e-111a-44c2-9ff2-6c2347444092\") "
Sep 15 10:59:40.341067 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.340861    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-model-cache" (OuterVolumeSpecName: "model-cache") pod "11e9fd4e-111a-44c2-9ff2-6c2347444092" (UID: "11e9fd4e-111a-44c2-9ff2-6c2347444092"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:59:40.341067 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.340960    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-home" (OuterVolumeSpecName: "home") pod "11e9fd4e-111a-44c2-9ff2-6c2347444092" (UID: "11e9fd4e-111a-44c2-9ff2-6c2347444092"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:59:40.341180 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.341070    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:40.341180 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.341090    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:40.341180 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.341116    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11e9fd4e-111a-44c2-9ff2-6c2347444092" (UID: "11e9fd4e-111a-44c2-9ff2-6c2347444092"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:59:40.342983 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.342959    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11e9fd4e-111a-44c2-9ff2-6c2347444092-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11e9fd4e-111a-44c2-9ff2-6c2347444092" (UID: "11e9fd4e-111a-44c2-9ff2-6c2347444092"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:59:40.343099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.343011    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-dshm" (OuterVolumeSpecName: "dshm") pod "11e9fd4e-111a-44c2-9ff2-6c2347444092" (UID: "11e9fd4e-111a-44c2-9ff2-6c2347444092"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:59:40.343099 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.343021    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e9fd4e-111a-44c2-9ff2-6c2347444092-kube-api-access-lp2kh" (OuterVolumeSpecName: "kube-api-access-lp2kh") pod "11e9fd4e-111a-44c2-9ff2-6c2347444092" (UID: "11e9fd4e-111a-44c2-9ff2-6c2347444092"). InnerVolumeSpecName "kube-api-access-lp2kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:59:40.442398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.442311    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e9fd4e-111a-44c2-9ff2-6c2347444092-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:40.442398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.442340    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp2kh\" (UniqueName: \"kubernetes.io/projected/11e9fd4e-111a-44c2-9ff2-6c2347444092-kube-api-access-lp2kh\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:40.442398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.442350    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:40.442398 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.442361    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11e9fd4e-111a-44c2-9ff2-6c2347444092-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:40.528140 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.528103    2584 generic.go:356] "Generic (PLEG): container finished" podID="11e9fd4e-111a-44c2-9ff2-6c2347444092" containerID="b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29" exitCode=0
Sep 15 10:59:40.528317 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.528168    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"
Sep 15 10:59:40.528317 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.528168    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg" event={"ID":"11e9fd4e-111a-44c2-9ff2-6c2347444092","Type":"ContainerDied","Data":"b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29"}
Sep 15 10:59:40.528317 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.528294    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg" event={"ID":"11e9fd4e-111a-44c2-9ff2-6c2347444092","Type":"ContainerDied","Data":"a112cd704d19c915fa343573de67bcd0a955b9c5aab29d57e7b963fb145f6d6f"}
Sep 15 10:59:40.528317 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.528317    2584 scope.go:117] "RemoveContainer" containerID="b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29"
Sep 15 10:59:40.537243 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.537222    2584 scope.go:117] "RemoveContainer" containerID="b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29"
Sep 15 10:59:40.537505 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:59:40.537484    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29\": container with ID starting with b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29 not found: ID does not exist" containerID="b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29"
Sep 15 10:59:40.537575 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.537519    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29"} err="failed to get container status \"b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29\": rpc error: code = NotFound desc = could not find container \"b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29\": container with ID starting with b599c27ef29ad3acb42f5312f267cd1dceed59dc16dca31f0bf35cbc50483c29 not found: ID does not exist"
Sep 15 10:59:40.552884 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.550628    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"]
Sep 15 10:59:40.557169 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:40.557149    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-eb5ed9e6-kserve-bfb4dbf4c-52fvg"]
Sep 15 10:59:41.760248 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:41.760170    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr" podUID="a256285e-2296-4c70-ba7c-3e4cdb479b63" containerName="main" containerID="cri-o://3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8" gracePeriod=45
Sep 15 10:59:41.784927 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:41.784899    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11e9fd4e-111a-44c2-9ff2-6c2347444092" path="/var/lib/kubelet/pods/11e9fd4e-111a-44c2-9ff2-6c2347444092/volumes"
Sep 15 10:59:42.004143 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.004118    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:59:42.054895 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.054816    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlsw7\" (UniqueName: \"kubernetes.io/projected/a256285e-2296-4c70-ba7c-3e4cdb479b63-kube-api-access-qlsw7\") pod \"a256285e-2296-4c70-ba7c-3e4cdb479b63\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") "
Sep 15 10:59:42.054895 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.054887    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-dshm\") pod \"a256285e-2296-4c70-ba7c-3e4cdb479b63\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") "
Sep 15 10:59:42.055129 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.054905    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-home\") pod \"a256285e-2296-4c70-ba7c-3e4cdb479b63\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") "
Sep 15 10:59:42.055129 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.054944    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a256285e-2296-4c70-ba7c-3e4cdb479b63-tls-certs\") pod \"a256285e-2296-4c70-ba7c-3e4cdb479b63\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") "
Sep 15 10:59:42.055129 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.054963    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-tmp-dir\") pod \"a256285e-2296-4c70-ba7c-3e4cdb479b63\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") "
Sep 15 10:59:42.055129 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.054994    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-model-cache\") pod \"a256285e-2296-4c70-ba7c-3e4cdb479b63\" (UID: \"a256285e-2296-4c70-ba7c-3e4cdb479b63\") "
Sep 15 10:59:42.055344 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.055240    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-home" (OuterVolumeSpecName: "home") pod "a256285e-2296-4c70-ba7c-3e4cdb479b63" (UID: "a256285e-2296-4c70-ba7c-3e4cdb479b63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:59:42.055344 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.055230    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a256285e-2296-4c70-ba7c-3e4cdb479b63" (UID: "a256285e-2296-4c70-ba7c-3e4cdb479b63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:59:42.055344 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.055277    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-model-cache" (OuterVolumeSpecName: "model-cache") pod "a256285e-2296-4c70-ba7c-3e4cdb479b63" (UID: "a256285e-2296-4c70-ba7c-3e4cdb479b63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:59:42.057103 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.057077    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a256285e-2296-4c70-ba7c-3e4cdb479b63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a256285e-2296-4c70-ba7c-3e4cdb479b63" (UID: "a256285e-2296-4c70-ba7c-3e4cdb479b63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:59:42.057207 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.057175    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-dshm" (OuterVolumeSpecName: "dshm") pod "a256285e-2296-4c70-ba7c-3e4cdb479b63" (UID: "a256285e-2296-4c70-ba7c-3e4cdb479b63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 10:59:42.057282 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.057258    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a256285e-2296-4c70-ba7c-3e4cdb479b63-kube-api-access-qlsw7" (OuterVolumeSpecName: "kube-api-access-qlsw7") pod "a256285e-2296-4c70-ba7c-3e4cdb479b63" (UID: "a256285e-2296-4c70-ba7c-3e4cdb479b63"). InnerVolumeSpecName "kube-api-access-qlsw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:59:42.156174 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.156142    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-dshm\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:42.156174 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.156169    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-home\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:42.156174 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.156178    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a256285e-2296-4c70-ba7c-3e4cdb479b63-tls-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:42.156411 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.156189    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-tmp-dir\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:42.156411 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.156198    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a256285e-2296-4c70-ba7c-3e4cdb479b63-model-cache\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:42.156411 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.156206    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlsw7\" (UniqueName: \"kubernetes.io/projected/a256285e-2296-4c70-ba7c-3e4cdb479b63-kube-api-access-qlsw7\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:42.540260 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.540227    2584 generic.go:356] "Generic (PLEG): container finished" podID="a256285e-2296-4c70-ba7c-3e4cdb479b63" containerID="3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8" exitCode=0
Sep 15 10:59:42.540451 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.540277    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"
Sep 15 10:59:42.540451 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.540313    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr" event={"ID":"a256285e-2296-4c70-ba7c-3e4cdb479b63","Type":"ContainerDied","Data":"3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8"}
Sep 15 10:59:42.540451 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.540357    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr" event={"ID":"a256285e-2296-4c70-ba7c-3e4cdb479b63","Type":"ContainerDied","Data":"bd210f8b541202b283ef5b08ab27dfd58e04b7eaaa02782a58158458e5840dca"}
Sep 15 10:59:42.540451 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.540378    2584 scope.go:117] "RemoveContainer" containerID="3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8"
Sep 15 10:59:42.549235 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.549219    2584 scope.go:117] "RemoveContainer" containerID="3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8"
Sep 15 10:59:42.549489 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:59:42.549471    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8\": container with ID starting with 3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8 not found: ID does not exist" containerID="3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8"
Sep 15 10:59:42.549533 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.549496    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8"} err="failed to get container status \"3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8\": rpc error: code = NotFound desc = could not find container \"3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8\": container with ID starting with 3cbf8154686cb03e4017dee0729b6d5e7e7b8bd48a89a0bed6ba640ada052db8 not found: ID does not exist"
Sep 15 10:59:42.561075 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.561054    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"]
Sep 15 10:59:42.566189 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.566168    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-eb5ed9e6-kserve-67559ddf75-969jr"]
Sep 15 10:59:42.753203 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.753168    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:59:42.958724 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.958628    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"]
Sep 15 10:59:42.959203 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.959129    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11e9fd4e-111a-44c2-9ff2-6c2347444092" containerName="main"
Sep 15 10:59:42.959203 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.959150    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="11e9fd4e-111a-44c2-9ff2-6c2347444092" containerName="main"
Sep 15 10:59:42.959203 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.959162    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a256285e-2296-4c70-ba7c-3e4cdb479b63" containerName="main"
Sep 15 10:59:42.959203 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.959170    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a256285e-2296-4c70-ba7c-3e4cdb479b63" containerName="main"
Sep 15 10:59:42.959433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.959254    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a256285e-2296-4c70-ba7c-3e4cdb479b63" containerName="main"
Sep 15 10:59:42.959433 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.959267    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="11e9fd4e-111a-44c2-9ff2-6c2347444092" containerName="main"
Sep 15 10:59:42.963749 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.963726    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:42.966062 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.966041    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 10:59:42.966182 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.966164    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 10:59:42.966985 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.966968    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 10:59:42.967075 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.967047    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dkr42\""
Sep 15 10:59:42.967142 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.967105    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 10:59:42.970242 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:42.970216    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"]
Sep 15 10:59:43.064211 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.064181    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vbl\" (UniqueName: \"kubernetes.io/projected/a0efc712-9929-490d-a3f6-99a80c4018bc-kube-api-access-69vbl\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.064211 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.064215    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0efc712-9929-490d-a3f6-99a80c4018bc-openshift-service-ca\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.064434 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.064250    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-maas-api-tls\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.064434 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.064325    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-metrics-certs\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.165437 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.165402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-maas-api-tls\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.165606 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.165447    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-metrics-certs\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.165606 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.165541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69vbl\" (UniqueName: \"kubernetes.io/projected/a0efc712-9929-490d-a3f6-99a80c4018bc-kube-api-access-69vbl\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.165606 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.165566    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0efc712-9929-490d-a3f6-99a80c4018bc-openshift-service-ca\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.166258 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.166237    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0efc712-9929-490d-a3f6-99a80c4018bc-openshift-service-ca\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.168098 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.168068    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-metrics-certs\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.168098 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.168095    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-maas-api-tls\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.173153 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.173127    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vbl\" (UniqueName: \"kubernetes.io/projected/a0efc712-9929-490d-a3f6-99a80c4018bc-kube-api-access-69vbl\") pod \"maas-api-5b77bbc99d-pntcz\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") " pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.275702 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.275663    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:43.546347 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.546201    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"]
Sep 15 10:59:43.548658 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:59:43.548619    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0efc712_9929_490d_a3f6_99a80c4018bc.slice/crio-8319189332b10b3c1c103e02fdad755d484d308467f31773a042fb4da11dac71 WatchSource:0}: Error finding container 8319189332b10b3c1c103e02fdad755d484d308467f31773a042fb4da11dac71: Status 404 returned error can't find the container with id 8319189332b10b3c1c103e02fdad755d484d308467f31773a042fb4da11dac71
Sep 15 10:59:43.549876 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.549856    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 10:59:43.784549 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:43.784516    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a256285e-2296-4c70-ba7c-3e4cdb479b63" path="/var/lib/kubelet/pods/a256285e-2296-4c70-ba7c-3e4cdb479b63/volumes"
Sep 15 10:59:44.552287 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:44.552250    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz" event={"ID":"a0efc712-9929-490d-a3f6-99a80c4018bc","Type":"ContainerStarted","Data":"7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a"}
Sep 15 10:59:44.552694 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:44.552309    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:44.552694 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:44.552328    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz" event={"ID":"a0efc712-9929-490d-a3f6-99a80c4018bc","Type":"ContainerStarted","Data":"8319189332b10b3c1c103e02fdad755d484d308467f31773a042fb4da11dac71"}
Sep 15 10:59:44.569369 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:44.569319    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz" podStartSLOduration=2.12821084 podStartE2EDuration="2.569305468s" totalImagesPullingTime="441.094628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:59:44.567068328 +0000 UTC m=+2503.261224037" watchObservedRunningTime="2026-09-15 10:59:44.569305468 +0000 UTC m=+2503.263461148"
Sep 15 10:59:50.356628 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.356590    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:59:50.555098 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.555055    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"]
Sep 15 10:59:50.555318 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.555279    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz" podUID="a0efc712-9929-490d-a3f6-99a80c4018bc" containerName="maas-api" containerID="cri-o://7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a" gracePeriod=30
Sep 15 10:59:50.561061 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.561040    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:50.587021 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.586998    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"]
Sep 15 10:59:50.590738 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.590718    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.598289 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.598262    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"]
Sep 15 10:59:50.627438 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.627358    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-openshift-service-ca\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.627438 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.627398    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-metrics-certs\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.627616 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.627498    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mspvc\" (UniqueName: \"kubernetes.io/projected/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-kube-api-access-mspvc\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.627616 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.627599    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-maas-api-tls\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.728431 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.728397    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-metrics-certs\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.728570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.728463    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mspvc\" (UniqueName: \"kubernetes.io/projected/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-kube-api-access-mspvc\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.728570 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.728532    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-maas-api-tls\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.728711 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.728571    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-openshift-service-ca\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.729302 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.729273    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-openshift-service-ca\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.731120 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.731092    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-metrics-certs\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.731428 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.731402    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-maas-api-tls\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.736600 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.736572    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mspvc\" (UniqueName: \"kubernetes.io/projected/bf8507f1-dfb5-4c8c-8700-59d5f88d367f-kube-api-access-mspvc\") pod \"maas-api-7dbbd6f894-grnqj\" (UID: \"bf8507f1-dfb5-4c8c-8700-59d5f88d367f\") " pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:50.797209 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.797180    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:50.929594 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.929511    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-metrics-certs\") pod \"a0efc712-9929-490d-a3f6-99a80c4018bc\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") "
Sep 15 10:59:50.929594 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.929569    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-maas-api-tls\") pod \"a0efc712-9929-490d-a3f6-99a80c4018bc\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") "
Sep 15 10:59:50.929826 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.929698    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69vbl\" (UniqueName: \"kubernetes.io/projected/a0efc712-9929-490d-a3f6-99a80c4018bc-kube-api-access-69vbl\") pod \"a0efc712-9929-490d-a3f6-99a80c4018bc\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") "
Sep 15 10:59:50.929826 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.929772    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0efc712-9929-490d-a3f6-99a80c4018bc-openshift-service-ca\") pod \"a0efc712-9929-490d-a3f6-99a80c4018bc\" (UID: \"a0efc712-9929-490d-a3f6-99a80c4018bc\") "
Sep 15 10:59:50.930280 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.930240    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0efc712-9929-490d-a3f6-99a80c4018bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0efc712-9929-490d-a3f6-99a80c4018bc" (UID: "a0efc712-9929-490d-a3f6-99a80c4018bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 10:59:50.931870 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.931847    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0efc712-9929-490d-a3f6-99a80c4018bc-kube-api-access-69vbl" (OuterVolumeSpecName: "kube-api-access-69vbl") pod "a0efc712-9929-490d-a3f6-99a80c4018bc" (UID: "a0efc712-9929-490d-a3f6-99a80c4018bc"). InnerVolumeSpecName "kube-api-access-69vbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 10:59:50.932129 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.932108    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0efc712-9929-490d-a3f6-99a80c4018bc" (UID: "a0efc712-9929-490d-a3f6-99a80c4018bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:59:50.932201 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.932129    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a0efc712-9929-490d-a3f6-99a80c4018bc" (UID: "a0efc712-9929-490d-a3f6-99a80c4018bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 10:59:50.935750 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:50.935727    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:51.031830 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.030936    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-metrics-certs\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:51.031830 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.031351    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0efc712-9929-490d-a3f6-99a80c4018bc-maas-api-tls\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:51.031830 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.031368    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69vbl\" (UniqueName: \"kubernetes.io/projected/a0efc712-9929-490d-a3f6-99a80c4018bc-kube-api-access-69vbl\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:51.031830 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.031401    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0efc712-9929-490d-a3f6-99a80c4018bc-openshift-service-ca\") on node \"ip-10-0-132-90.ec2.internal\" DevicePath \"\""
Sep 15 10:59:51.207270 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.207193    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"]
Sep 15 10:59:51.210298 ip-10-0-132-90 kubenswrapper[2584]: W0915 10:59:51.210269    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8507f1_dfb5_4c8c_8700_59d5f88d367f.slice/crio-86c8f1170604a97f42c4f5c7df2c45f83b077fa75a250cb7c0bdfe93a1a19552 WatchSource:0}: Error finding container 86c8f1170604a97f42c4f5c7df2c45f83b077fa75a250cb7c0bdfe93a1a19552: Status 404 returned error can't find the container with id 86c8f1170604a97f42c4f5c7df2c45f83b077fa75a250cb7c0bdfe93a1a19552
Sep 15 10:59:51.585097 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.585067    2584 generic.go:356] "Generic (PLEG): container finished" podID="a0efc712-9929-490d-a3f6-99a80c4018bc" containerID="7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a" exitCode=0
Sep 15 10:59:51.585475 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.585133    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"
Sep 15 10:59:51.585475 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.585150    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz" event={"ID":"a0efc712-9929-490d-a3f6-99a80c4018bc","Type":"ContainerDied","Data":"7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a"}
Sep 15 10:59:51.585475 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.585206    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz" event={"ID":"a0efc712-9929-490d-a3f6-99a80c4018bc","Type":"ContainerDied","Data":"8319189332b10b3c1c103e02fdad755d484d308467f31773a042fb4da11dac71"}
Sep 15 10:59:51.585475 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.585229    2584 scope.go:117] "RemoveContainer" containerID="7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a"
Sep 15 10:59:51.586447 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.586429    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj" event={"ID":"bf8507f1-dfb5-4c8c-8700-59d5f88d367f","Type":"ContainerStarted","Data":"86c8f1170604a97f42c4f5c7df2c45f83b077fa75a250cb7c0bdfe93a1a19552"}
Sep 15 10:59:51.600357 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.600335    2584 scope.go:117] "RemoveContainer" containerID="7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a"
Sep 15 10:59:51.600685 ip-10-0-132-90 kubenswrapper[2584]: E0915 10:59:51.600660    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a\": container with ID starting with 7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a not found: ID does not exist" containerID="7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a"
Sep 15 10:59:51.600765 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.600693    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a"} err="failed to get container status \"7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a\": rpc error: code = NotFound desc = could not find container \"7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a\": container with ID starting with 7d32cd34d1055b7cc063bba9d6d91c92c57d7c505f0a637383c8594294c1262a not found: ID does not exist"
Sep 15 10:59:51.608064 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.608042    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"]
Sep 15 10:59:51.612258 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.612234    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b77bbc99d-pntcz"]
Sep 15 10:59:51.786910 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:51.786876    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0efc712-9929-490d-a3f6-99a80c4018bc" path="/var/lib/kubelet/pods/a0efc712-9929-490d-a3f6-99a80c4018bc/volumes"
Sep 15 10:59:52.592027 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:52.591981    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj" event={"ID":"bf8507f1-dfb5-4c8c-8700-59d5f88d367f","Type":"ContainerStarted","Data":"fb5ebdf6cf57db11e06688e65c446cb19eb56c033d908ac737446188afe72276"}
Sep 15 10:59:52.592488 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:52.592118    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 10:59:52.609356 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:52.609267    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj" podStartSLOduration=2.263547583 podStartE2EDuration="2.609250144s" totalImagesPullingTime="345.702561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 10:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 10:59:52.607710515 +0000 UTC m=+2511.301866191" watchObservedRunningTime="2026-09-15 10:59:52.609250144 +0000 UTC m=+2511.303405827"
Sep 15 10:59:57.256375 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:57.256294    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 10:59:58.602972 ip-10-0-132-90 kubenswrapper[2584]: I0915 10:59:58.602939    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7dbbd6f894-grnqj"
Sep 15 11:00:03.347823 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:03.347762    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 11:00:10.158312 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:10.158270    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-k659j"]
Sep 15 11:00:12.516812 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:12.516771    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68ccdd49b4-pqt29_fda4c709-2973-480b-8a9b-7cfe35c77070/authorino/0.log"
Sep 15 11:00:17.840843 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:17.840787    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z6prd_128887f8-37c7-4036-ac64-5644ffce6f38/manager/0.log"
Sep 15 11:00:18.427586 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:18.427559    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6699d7bd9c-46dfj_0556b2fa-e5ad-4e59-b859-7c3a133d93cf/copy-manifests/0.log"
Sep 15 11:00:18.439517 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:18.439488    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6699d7bd9c-46dfj_0556b2fa-e5ad-4e59-b859-7c3a133d93cf/manager/0.log"
Sep 15 11:00:19.091793 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:19.091762    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7dbbd6f894-grnqj_bf8507f1-dfb5-4c8c-8700-59d5f88d367f/maas-api/0.log"
Sep 15 11:00:20.243372 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:20.243341    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68ccdd49b4-pqt29_fda4c709-2973-480b-8a9b-7cfe35c77070/authorino/0.log"
Sep 15 11:00:20.570764 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:20.570688    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-dhj2d_b677254b-e134-46bb-9ea4-a6c2faeaa047/kuadrant-console-plugin/0.log"
Sep 15 11:00:20.793231 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:20.793192    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-k659j_7239e98d-5135-49de-ac86-9979c8e37280/limitador/0.log"
Sep 15 11:00:21.554869 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:21.554834    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58df565b49-s7f5p_f1e3360c-f192-4adf-a2f6-815658a66fc5/kube-auth-proxy/0.log"
Sep 15 11:00:22.637093 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:22.637048    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5wn4g_2877d7e7-14ea-4a16-bee1-08281728d61f/main/0.log"
Sep 15 11:00:22.741161 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:22.741132    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsdns4_1dbfaf81-5967-4d80-a60e-b069ccb75342/main/0.log"
Sep 15 11:00:22.959919 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:22.959795    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-b68lp_71dbe841-0517-42f6-bdea-c0ddbf9ec3a7/main/0.log"
Sep 15 11:00:31.297730 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:31.297686    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-txhqg_992c0c97-0af1-4197-82ca-e4c193a935f6/global-pull-secret-syncer/0.log"
Sep 15 11:00:31.403732 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:31.403697    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n8bbj_a0582a9b-c2c1-437e-99a6-0a907acb9b35/konnectivity-agent/0.log"
Sep 15 11:00:31.423293 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:31.423226    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-90.ec2.internal_18620f5e88bb272825dcdf930e2ff0ad/haproxy/0.log"
Sep 15 11:00:36.071007 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:36.070970    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68ccdd49b4-pqt29_fda4c709-2973-480b-8a9b-7cfe35c77070/authorino/0.log"
Sep 15 11:00:36.156655 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:36.156629    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-dhj2d_b677254b-e134-46bb-9ea4-a6c2faeaa047/kuadrant-console-plugin/0.log"
Sep 15 11:00:36.248579 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:36.248546    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-k659j_7239e98d-5135-49de-ac86-9979c8e37280/limitador/0.log"
Sep 15 11:00:38.003504 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.003470    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-w997q_d708ecc6-3d6a-4921-a47d-3a17f0de0b1a/kube-state-metrics/0.log"
Sep 15 11:00:38.025887 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.025855    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-w997q_d708ecc6-3d6a-4921-a47d-3a17f0de0b1a/kube-rbac-proxy-main/0.log"
Sep 15 11:00:38.047661 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.047634    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-w997q_d708ecc6-3d6a-4921-a47d-3a17f0de0b1a/kube-rbac-proxy-self/0.log"
Sep 15 11:00:38.074942 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.074920    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5fc745c9c8-5v7b4_83738ffd-b578-426f-a6a4-b66e88b01b7f/metrics-server/0.log"
Sep 15 11:00:38.138246 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.138214    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6bzth_796cc65d-7664-4e9a-ba6f-7c0da2ac0b64/node-exporter/0.log"
Sep 15 11:00:38.159651 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.159628    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6bzth_796cc65d-7664-4e9a-ba6f-7c0da2ac0b64/kube-rbac-proxy/0.log"
Sep 15 11:00:38.182394 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.182373    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6bzth_796cc65d-7664-4e9a-ba6f-7c0da2ac0b64/init-textfile/0.log"
Sep 15 11:00:38.383437 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.383410    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-k95kh_0ecc33dd-55c1-491c-918a-5d52916e5535/kube-rbac-proxy-main/0.log"
Sep 15 11:00:38.405312 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.405283    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-k95kh_0ecc33dd-55c1-491c-918a-5d52916e5535/kube-rbac-proxy-self/0.log"
Sep 15 11:00:38.430215 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.430184    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-k95kh_0ecc33dd-55c1-491c-918a-5d52916e5535/openshift-state-metrics/0.log"
Sep 15 11:00:38.493587 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.493558    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1/prometheus/0.log"
Sep 15 11:00:38.517968 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.517939    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1/config-reloader/0.log"
Sep 15 11:00:38.543570 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.543542    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1/thanos-sidecar/0.log"
Sep 15 11:00:38.564455 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.564434    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1/kube-rbac-proxy-web/0.log"
Sep 15 11:00:38.592136 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.592064    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1/kube-rbac-proxy/0.log"
Sep 15 11:00:38.615254 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.615230    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1/kube-rbac-proxy-thanos/0.log"
Sep 15 11:00:38.647033 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:38.647010    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c53a75d-78ad-41af-8d4a-3b4f03fe0ec1/init-config-reloader/0.log"
Sep 15 11:00:41.661192 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:41.661158    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-fm6hc_8f151224-d92a-49a4-90e7-f4f2448dda7a/download-server/0.log"
Sep 15 11:00:42.284275 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.284237    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"]
Sep 15 11:00:42.284584 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.284572    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0efc712-9929-490d-a3f6-99a80c4018bc" containerName="maas-api"
Sep 15 11:00:42.284631 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.284586    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0efc712-9929-490d-a3f6-99a80c4018bc" containerName="maas-api"
Sep 15 11:00:42.284672 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.284660    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0efc712-9929-490d-a3f6-99a80c4018bc" containerName="maas-api"
Sep 15 11:00:42.287441 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.287424    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.289750 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.289726    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2wphs\"/\"openshift-service-ca.crt\""
Sep 15 11:00:42.290028 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.290012    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2wphs\"/\"kube-root-ca.crt\""
Sep 15 11:00:42.290810 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.290778    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2wphs\"/\"default-dockercfg-4b7wp\""
Sep 15 11:00:42.296485 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.296461    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"]
Sep 15 11:00:42.405304 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.405273    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-sys\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.405304 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.405312    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-lib-modules\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.405534 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.405350    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-podres\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.405534 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.405365    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-proc\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.405534 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.405410    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsj27\" (UniqueName: \"kubernetes.io/projected/4eb2296e-3719-42e7-944f-de254e762b29-kube-api-access-vsj27\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.506073 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.506031    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-sys\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.506263 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.506079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-lib-modules\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.506263 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.506138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-podres\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.506263 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.506163    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-sys\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.506263 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.506163    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-proc\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.506263 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.506258    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsj27\" (UniqueName: \"kubernetes.io/projected/4eb2296e-3719-42e7-944f-de254e762b29-kube-api-access-vsj27\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.506481 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.506281    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-lib-modules\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.506481 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.506302    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-proc\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.506481 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.506327    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4eb2296e-3719-42e7-944f-de254e762b29-podres\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.514526 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.514502    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsj27\" (UniqueName: \"kubernetes.io/projected/4eb2296e-3719-42e7-944f-de254e762b29-kube-api-access-vsj27\") pod \"perf-node-gather-daemonset-bxgrp\" (UID: \"4eb2296e-3719-42e7-944f-de254e762b29\") " pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.598614 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.598529    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:42.870161 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:42.870083    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"]
Sep 15 11:00:42.872976 ip-10-0-132-90 kubenswrapper[2584]: W0915 11:00:42.872946    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4eb2296e_3719_42e7_944f_de254e762b29.slice/crio-dce80d556f3435165c3c7cb30bd55c676191f43987f47391bd0c5a870a900cee WatchSource:0}: Error finding container dce80d556f3435165c3c7cb30bd55c676191f43987f47391bd0c5a870a900cee: Status 404 returned error can't find the container with id dce80d556f3435165c3c7cb30bd55c676191f43987f47391bd0c5a870a900cee
Sep 15 11:00:43.717705 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:43.717664    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gcxvj_42c7db80-61f8-46e5-8458-3db6c66a56c3/dns/0.log"
Sep 15 11:00:43.737654 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:43.737632    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gcxvj_42c7db80-61f8-46e5-8458-3db6c66a56c3/kube-rbac-proxy/0.log"
Sep 15 11:00:43.811869 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:43.811834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp" event={"ID":"4eb2296e-3719-42e7-944f-de254e762b29","Type":"ContainerStarted","Data":"1334e4c5994ad7ecb7a01b669a51161a990c75ddf7ee9130bd429611d7520ae5"}
Sep 15 11:00:43.812038 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:43.811889    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp" event={"ID":"4eb2296e-3719-42e7-944f-de254e762b29","Type":"ContainerStarted","Data":"dce80d556f3435165c3c7cb30bd55c676191f43987f47391bd0c5a870a900cee"}
Sep 15 11:00:43.812038 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:43.811917    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:43.825963 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:43.825770    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp" podStartSLOduration=1.825753738 podStartE2EDuration="1.825753738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:00:43.824891244 +0000 UTC m=+2562.519046924" watchObservedRunningTime="2026-09-15 11:00:43.825753738 +0000 UTC m=+2562.519909418"
Sep 15 11:00:43.873943 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:43.873908    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q4zj5_b1e9ea1a-0387-4294-a760-df3980219768/dns-node-resolver/0.log"
Sep 15 11:00:44.438601 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:44.438569    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79dd5bfd77-hrtl6_f1c396c5-e224-478f-a09d-6857eee1ae0e/registry/0.log"
Sep 15 11:00:44.530698 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:44.530670    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7vfnf_72c44564-2d7f-47fd-8567-d8793ed885bd/node-ca/0.log"
Sep 15 11:00:46.010932 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:46.010903    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58df565b49-s7f5p_f1e3360c-f192-4adf-a2f6-815658a66fc5/kube-auth-proxy/0.log"
Sep 15 11:00:47.044553 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:47.044520    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hwtxj_146f0ea1-aecd-451f-b1f1-8bc242b439d8/serve-healthcheck-canary/0.log"
Sep 15 11:00:47.632775 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:47.632747    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4qg4r_89c8a64a-26f9-45b5-a4a8-c1d06fcffad1/kube-rbac-proxy/0.log"
Sep 15 11:00:47.663367 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:47.663341    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4qg4r_89c8a64a-26f9-45b5-a4a8-c1d06fcffad1/exporter/0.log"
Sep 15 11:00:47.683419 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:47.683385    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4qg4r_89c8a64a-26f9-45b5-a4a8-c1d06fcffad1/extractor/0.log"
Sep 15 11:00:49.797864 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:49.797834    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z6prd_128887f8-37c7-4036-ac64-5644ffce6f38/manager/0.log"
Sep 15 11:00:49.826636 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:49.826607    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2wphs/perf-node-gather-daemonset-bxgrp"
Sep 15 11:00:50.363153 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:50.363120    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6699d7bd9c-46dfj_0556b2fa-e5ad-4e59-b859-7c3a133d93cf/manager/0.log"
Sep 15 11:00:50.379285 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:50.379258    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6699d7bd9c-46dfj_0556b2fa-e5ad-4e59-b859-7c3a133d93cf/copy-manifests/0.log"
Sep 15 11:00:51.920301 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:51.920217    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-85d6978c87-4hl5x_f4eb586c-020c-48ac-9040-9eec68995175/manager/0.log"
Sep 15 11:00:52.786116 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:52.786085    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58df565b49-s7f5p_f1e3360c-f192-4adf-a2f6-815658a66fc5/kube-auth-proxy/0.log"
Sep 15 11:00:54.916305 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:54.916269    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58df565b49-s7f5p_f1e3360c-f192-4adf-a2f6-815658a66fc5/kube-auth-proxy/0.log"
Sep 15 11:00:58.700273 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:58.700244    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgms7_33162a7a-bbbc-475d-b186-63e2b94656fe/kube-multus-additional-cni-plugins/0.log"
Sep 15 11:00:58.720444 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:58.720412    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgms7_33162a7a-bbbc-475d-b186-63e2b94656fe/egress-router-binary-copy/0.log"
Sep 15 11:00:58.743451 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:58.743424    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgms7_33162a7a-bbbc-475d-b186-63e2b94656fe/cni-plugins/0.log"
Sep 15 11:00:58.763581 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:58.763556    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgms7_33162a7a-bbbc-475d-b186-63e2b94656fe/bond-cni-plugin/0.log"
Sep 15 11:00:58.783752 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:58.783733    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgms7_33162a7a-bbbc-475d-b186-63e2b94656fe/routeoverride-cni/0.log"
Sep 15 11:00:58.804291 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:58.804267    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgms7_33162a7a-bbbc-475d-b186-63e2b94656fe/whereabouts-cni-bincopy/0.log"
Sep 15 11:00:58.824256 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:58.824239    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgms7_33162a7a-bbbc-475d-b186-63e2b94656fe/whereabouts-cni/0.log"
Sep 15 11:00:59.105885 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:59.105859    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zf2rw_c8246177-ac46-487b-9853-e979365daa1f/kube-multus/0.log"
Sep 15 11:00:59.127683 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:59.127658    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bftt2_6992d04d-f1f5-4521-a0da-27555d562676/network-metrics-daemon/0.log"
Sep 15 11:00:59.145963 ip-10-0-132-90 kubenswrapper[2584]: I0915 11:00:59.145935    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bftt2_6992d04d-f1f5-4521-a0da-27555d562676/kube-rbac-proxy/0.log"