Sep 09 20:48:43.235443 ip-10-0-136-121 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:48:43.625092 ip-10-0-136-121 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:48:43.625092 ip-10-0-136-121 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:48:43.625092 ip-10-0-136-121 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:48:43.625092 ip-10-0-136-121 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:48:43.625092 ip-10-0-136-121 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:48:43.626711 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.626633    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:48:43.629645 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629630    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:43.629645 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629645    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629649    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629652    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629656    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629659    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629662    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629666    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629669    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629671    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629674    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629677    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629680    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629683    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629692    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629694    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629697    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629700    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629702    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629705    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629708    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:43.629710 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629711    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629713    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629716    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629719    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629722    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629725    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629729    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629733    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629736    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629739    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629741    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629744    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629747    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629750    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629753    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629756    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629758    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629762    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629766    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:43.630200 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629770    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629773    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629775    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629778    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629780    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629785    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629789    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629792    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629794    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629797    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629800    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629802    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629805    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629807    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629810    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629814    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629817    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629820    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629822    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:43.630659 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629825    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629827    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629830    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629833    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629837    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629839    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629842    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629845    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629847    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629850    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629852    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629855    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629858    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629861    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629864    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629866    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629869    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629871    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629874    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:43.631155 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629876    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629880    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629882    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629884    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629887    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629889    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629892    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629894    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629897    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629900    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629903    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629906    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629908    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629911    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629913    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629916    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629918    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.629921    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.630420    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.630426    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:43.631639 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.630429    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631137    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631143    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631147    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631149    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631152    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631156    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631159    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631162    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631165    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631168    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631187    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631189    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631192    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631195    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631197    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631200    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631205    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631208    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631211    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:43.632120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631215    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631218    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631221    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631224    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631226    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631229    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631232    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631235    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631237    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631240    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631242    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631246    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631250    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631252    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631255    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631258    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631261    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631263    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631266    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631269    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:43.632652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631271    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631274    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631286    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631291    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631294    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631297    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631300    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631303    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631307    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631310    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631313    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631315    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631318    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631321    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631324    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631326    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631329    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631332    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631335    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631337    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:43.633142 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631340    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631342    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631345    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631347    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631350    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631352    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631355    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631357    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631360    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631362    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631365    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631367    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631370    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631372    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631375    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631379    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631383    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631387    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631390    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:43.633656 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631392    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631395    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631398    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631401    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631403    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631406    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631409    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631412    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631414    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631417    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631419    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631422    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631425    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631427    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631430    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631433    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631435    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631437    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631440    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631443    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:43.634120 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631445    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631885    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631889    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631892    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631895    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631903    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631906    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631909    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631912    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631914    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631917    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631920    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631929    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631932    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631934    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631937    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631939    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631942    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631945    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631947    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:43.634625 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631952    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631956    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631959    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631962    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631965    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631967    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631970    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631973    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631975    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631978    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631980    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631983    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631985    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631987    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631990    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631993    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631995    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.631998    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632006    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632008    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:43.635110 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632011    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632014    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632017    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632021    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632023    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632026    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632029    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632032    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632034    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632037    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632039    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632042    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632045    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632047    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632050    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632052    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632056    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632060    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632063    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:43.635614 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632065    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632068    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632070    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632073    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632076    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632078    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632081    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632083    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632086    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632088    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632091    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632093    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632097    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632099    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632102    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632104    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632106    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632109    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632112    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632115    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:43.636109 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632117    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632120    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632123    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632126    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632128    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632131    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632134    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632136    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632139    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632141    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632144    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632146    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632149    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632152    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632154    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632159    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632162    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632164    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632167    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:43.636607 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632183    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632186    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632263    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632295    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632302    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632306    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632310    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632314    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632318    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632322    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632325    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632330    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632334    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632337    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632340    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632343    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632346    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632349    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632351    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632354    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632358    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632361    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632364    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632368    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:48:43.637081 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632371    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632375    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632378    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632381    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632385    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632388    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632390    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632393    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632396    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632399    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632403    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632406    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632409    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632412    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632415    2578 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632418    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632423    2578 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632426    2578 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632429    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632434    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632437    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632441    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632443    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632447    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632450    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:48:43.637681 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632452    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632455    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632458    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632461    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632464    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632467    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632470    2578 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632474    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632477    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632480    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632483    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632486    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632489    2578 flags.go:64] FLAG: --help="false"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632491    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-121.ec2.internal"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632495    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632498    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632500    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632504    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632507    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632510    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632513    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632516    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632518    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:48:43.638283 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632527    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632530    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632533    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632538    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632540    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632543    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632546    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632550    2578 flags.go:64] FLAG: --lock-file=""
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632552    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632555    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632558    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632563    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632566    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632569    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632572    2578 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632575    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632579    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632581    2578 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632584    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632589    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632591    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632595    2578 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632598    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632601    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632604    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:48:43.638870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632607    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632610    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632613    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632616    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632622    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632625    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632628    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632632    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632635    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632641    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632644    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632648    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632651    2578 flags.go:64] FLAG: --port="10250"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632654    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632657    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-036d3a8fd2b3bb5e4"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632660    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632664    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632667    2578 flags.go:64] FLAG: --register-node="true"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632670    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632673    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632677    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632680    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632683    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632687    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632690    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:48:43.639572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632693    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632696    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632699    2578 flags.go:64] FLAG: --runonce="false"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632701    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632705    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632708    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632710    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632713    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632716    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632719    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632722    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632725    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632728    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632731    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632734    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632737    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632740    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632743    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632749    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632752    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632755    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632759    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632762    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632765    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632768    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:48:43.640276 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632771    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632774    2578 flags.go:64] FLAG: --v="2"
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632779    2578 flags.go:64] FLAG: --version="false"
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632782    2578 flags.go:64] FLAG: --vmodule=""
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632786    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.632789    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632918    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632923    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632926    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632929    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632934    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632937    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632941    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632944    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632947    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632949    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632952    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632954    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632957    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632960    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632962    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:43.640867 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632965    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632967    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632970    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632973    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632977    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632980    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632983    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632986    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632989    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632991    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632994    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.632997    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633000    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633004    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633015    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633018    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633022    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633026    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633029    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:43.641411 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633032    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633035    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633038    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633041    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633044    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633047    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633049    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633052    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633055    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633058    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633062    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633071    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633075    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633077    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633080    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633082    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633085    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633090    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633092    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633095    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:43.641877 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633098    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633100    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633103    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633106    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633109    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633111    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633115    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633118    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633120    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633122    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633125    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633128    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633130    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633133    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633135    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633138    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633140    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633143    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633146    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:43.642375 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633148    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633151    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633154    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633157    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633160    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633162    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633164    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633168    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633182    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633185    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633189    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633192    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633194    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633197    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633199    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633202    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633205    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633207    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633210    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633214    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:43.642902 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633216    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633219    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633221    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633224    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633226    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.633229    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.633728    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.640433    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.640447    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.640461    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.640467    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640521    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640526    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640529    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640532    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:43.643535 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640535    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640538    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640541    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640544    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640546    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640549    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640551    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640554    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640557    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640560    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640563    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640565    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640568    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640571    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640573    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640576    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640580    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640584    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640586    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640589    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:43.643946 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640591    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640594    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640597    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640599    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640602    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640605    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640608    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640611    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640615    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640619    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640622    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640625    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640629    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640632    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640635    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640638    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640641    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640643    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640646    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:43.644518 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640648    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640651    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640654    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640657    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640659    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640662    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640665    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640667    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640670    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640672    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640675    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640677    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640680    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640683    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640685    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640688    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640690    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640693    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640696    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640698    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:43.644983 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640701    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640704    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640706    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640709    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640711    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640714    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640717    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640720    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640722    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640725    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640727    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640730    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640732    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640735    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640737    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640740    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640743    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640745    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640748    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640750    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:43.645497 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640753    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640755    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640758    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640761    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640763    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640766    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640768    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640771    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640773    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640775    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640778    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640781    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640784    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640786    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640789    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:43.645980 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640792    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.640797    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640909    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640913    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640916    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640918    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640921    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640924    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640926    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640928    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640931    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640934    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640936    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640939    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640942    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:43.646365 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640944    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640947    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640951    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640954    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640957    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640960    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640962    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640965    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640967    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640971    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640975    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640978    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640981    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640984    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640987    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640990    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640993    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640996    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.640999    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:43.646724 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641002    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641005    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641007    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641010    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641012    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641015    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641017    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641020    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641022    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641025    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641027    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641030    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641033    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641036    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641038    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641041    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641044    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641047    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641049    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:43.647194 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641052    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641054    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641057    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641059    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641061    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641072    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641075    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641078    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641081    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641083    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641086    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641088    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641091    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641095    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641098    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641101    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641104    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641106    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641109    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:43.647652 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641111    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641114    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641116    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641119    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641121    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641124    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641126    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641129    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641132    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641135    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641137    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641140    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641142    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641145    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641147    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641150    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641153    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641155    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641157    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641160    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:43.648102 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641163    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641165    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641167    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641184    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641188    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641190    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641193    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641195    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:43.641198    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.641203    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.641298    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.645156    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.646070    2578 server.go:1019] "Starting client certificate rotation"
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.646166    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:48:43.648671 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.647061    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:48:43.649390 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.649377    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:48:43.653138 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.653123    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:48:43.660814 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.660796    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:48:43.665006 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.664986    2578 log.go:25] "Validated CRI v1 image API"
Sep 09 20:48:43.665903 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.665890    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:48:43.668252 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.668229    2578 fs.go:135] Filesystem UUIDs: map[4f4044f2-fdb3-434a-8506-b05952ac2534:/dev/nvme0n1p3 7412f877-38c1-4e14-9f55-0aa0358a945d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 09 20:48:43.668311 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.668251    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:48:43.673975 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.673811    2578 manager.go:216] Machine: {Timestamp:2026-09-09 20:48:43.67264426 +0000 UTC m=+0.327424409 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105596 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24980461872edf82790b26b1d27f72 SystemUUID:ec249804-6187-2edf-8279-0b26b1d27f72 BootID:9e9ad854-c25b-493e-bf78-c711406d2a29 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9c:99:64:8f:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9c:99:64:8f:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:8e:bf:30:c4:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:48:43.673975 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.673965    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:48:43.674139 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.673993    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:48:43.674139 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.674075    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:48:43.677747 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.677719    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:48:43.677873 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.677750    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:48:43.678410 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.678394    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:48:43.678410 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.678412    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:48:43.678511 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.678421    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:48:43.678997 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.678987    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:48:43.680205 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.680196    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:48:43.683800 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.683789    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:48:43.683913 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.683904    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:48:43.683943 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.683932    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:48:43.683973 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.683942    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:48:43.683973 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.683957    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:48:43.683973 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.683967    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:48:43.684060 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.683975    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:48:43.684376 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.684356    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:48:43.684462 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.684386    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:48:43.687928 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.687902    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:48:43.691970 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.691954    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:48:43.692034 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.691985    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:48:43.692134 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692124    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:48:43.692165 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692140    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:48:43.692165 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692146    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:48:43.692165 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692151    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:48:43.692165 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692159    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:48:43.692165 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692164    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:48:43.692576 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692564    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:48:43.692612 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692578    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:48:43.692612 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692586    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:48:43.692612 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692591    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:48:43.692612 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692600    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:48:43.692612 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.692608    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:48:43.693363 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.693351    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:48:43.693401 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.693365    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:48:43.696814 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.696785    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:48:43.696899 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.696830    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:48:43.697111 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.697097    2578 server.go:1268] "Started kubelet"
Sep 09 20:48:43.697191 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.697136    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:48:43.697255 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.697168    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:48:43.697255 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.697201    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:48:43.697345 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.697260    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:48:43.697846 ip-10-0-136-121 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:48:43.698021 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.698006    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s87bv"
Sep 09 20:48:43.698341 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.698324    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:48:43.700775 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.699660    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:48:43.703028 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.703004    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:48:43.703101 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.703078    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:48:43.703156 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.703107    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:48:43.703214 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.703152    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:43.703677 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.703627    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:48:43.703677 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.703652    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:48:43.703792 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.703627    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:48:43.703847 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.703805    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:48:43.703847 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.703812    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:48:43.703932 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.703847    2578 factory.go:55] Registering systemd factory
Sep 09 20:48:43.703932 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.703862    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:48:43.704049 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.704036    2578 factory.go:153] Registering CRI-O factory
Sep 09 20:48:43.704049 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.704047    2578 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:48:43.704130 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.704097    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:48:43.704130 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.704114    2578 factory.go:103] Registering Raw factory
Sep 09 20:48:43.704130 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.704126    2578 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:48:43.704676 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.704657    2578 manager.go:349] Starting recovery of all containers
Sep 09 20:48:43.704825 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.704800    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:48:43.704912 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.704894    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:48:43.705807 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.704780    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-121.ec2.internal.18d3c220eec87c54  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-121.ec2.internal,UID:ip-10-0-136-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-121.ec2.internal,},FirstTimestamp:2026-09-09 20:48:43.697077332 +0000 UTC m=+0.351857482,LastTimestamp:2026-09-09 20:48:43.697077332 +0000 UTC m=+0.351857482,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-121.ec2.internal,}"
Sep 09 20:48:43.706213 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.706188    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s87bv"
Sep 09 20:48:43.715119 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.715100    2578 manager.go:354] Recovery completed
Sep 09 20:48:43.722526 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.722513    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:43.725470 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.725454    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:43.725541 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.725483    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:43.725541 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.725495    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:43.725975 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.725962    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:48:43.725975 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.725974    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:48:43.726075 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.725980    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:43.726075 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.725988    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:48:43.726682 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.726665    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:43.726761 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.726691    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:43.726761 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.726701    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:43.727870 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.727855    2578 policy_none.go:49] "None policy: Start"
Sep 09 20:48:43.727929 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.727872    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:48:43.727929 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.727883    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:48:43.729782 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.729771    2578 policy_none.go:47] "Start"
Sep 09 20:48:43.756745 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.756730    2578 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:48:43.776036 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.756769    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:48:43.776036 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.756782    2578 server.go:87] "Starting device plugin registration server"
Sep 09 20:48:43.776036 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.757005    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:48:43.776036 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.757017    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:48:43.776036 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.757115    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:48:43.776036 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.757204    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:48:43.776036 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.757212    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:48:43.776036 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.757736    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:48:43.776036 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.757774    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:43.833475 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.833447    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:48:43.834630 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.834609    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:48:43.834718 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.834656    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:48:43.834718 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.834678    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:48:43.834829 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.834715    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:48:43.837161 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.837140    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:48:43.858108 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.858078    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:43.859040 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.859015    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:43.859113 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.859051    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:43.859113 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.859065    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:43.859113 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.859089    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-121.ec2.internal"
Sep 09 20:48:43.865223 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.865209    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-121.ec2.internal"
Sep 09 20:48:43.865291 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.865229    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-121.ec2.internal\": node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:43.880493 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.880427    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:43.935037 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.934999    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"]
Sep 09 20:48:43.935220 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.935100    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:43.936205 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.936187    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:43.936273 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.936228    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:43.936273 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.936242    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:43.938506 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.938493    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:43.938627 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.938611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:43.938673 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.938645    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:43.939241 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.939223    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:43.939355 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.939255    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:43.939355 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.939271    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:43.939355 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.939231    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:43.939355 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.939318    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:43.939355 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.939330    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:43.941331 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.941313    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:43.941416 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.941340    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:43.941980 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.941965    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:43.942053 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.941988    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:43.942053 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:43.942001    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:43.968850 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.968829    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-121.ec2.internal\" not found" node="ip-10-0-136-121.ec2.internal"
Sep 09 20:48:43.973568 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.973551    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-121.ec2.internal\" not found" node="ip-10-0-136-121.ec2.internal"
Sep 09 20:48:43.980749 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:43.980734    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:44.006090 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.006074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/97370ad8a194f963a134d4e162f93aef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"97370ad8a194f963a134d4e162f93aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.006188 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.006099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97370ad8a194f963a134d4e162f93aef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"97370ad8a194f963a134d4e162f93aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.006188 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.006115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a618664de1a81cb8ba66310981e215f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-121.ec2.internal\" (UID: \"1a618664de1a81cb8ba66310981e215f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.081608 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:44.081584    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:44.106918 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.106898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97370ad8a194f963a134d4e162f93aef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"97370ad8a194f963a134d4e162f93aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.106986 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.106923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a618664de1a81cb8ba66310981e215f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-121.ec2.internal\" (UID: \"1a618664de1a81cb8ba66310981e215f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.106986 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.106939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/97370ad8a194f963a134d4e162f93aef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"97370ad8a194f963a134d4e162f93aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.106986 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.106974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a618664de1a81cb8ba66310981e215f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-121.ec2.internal\" (UID: \"1a618664de1a81cb8ba66310981e215f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.106986 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.106977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97370ad8a194f963a134d4e162f93aef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"97370ad8a194f963a134d4e162f93aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.107106 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.107015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/97370ad8a194f963a134d4e162f93aef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"97370ad8a194f963a134d4e162f93aef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.182241 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:44.182158    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:44.271883 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.271840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.276802 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.276680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.283167 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:44.283149    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:44.389261 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:44.389217    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:44.489449 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:44.489358    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:44.536225 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.536198    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:48:44.590578 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:44.590550    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:44.646756 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.646729    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:48:44.647183 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.646870    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:48:44.647183 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.646900    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:48:44.691238 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:44.691209    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 09 20:48:44.709143 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.709103    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:43:44 +0000 UTC" deadline="2028-02-14 22:33:56.285286621 +0000 UTC"
Sep 09 20:48:44.709143 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.709138    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12553h45m11.576153176s"
Sep 09 20:48:44.746102 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.745897    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:48:44.803805 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.803781    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.813720 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.813701    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:48:44.815591 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.815577    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 09 20:48:44.826157 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.826140    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:48:44.918304 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:44.918272    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97370ad8a194f963a134d4e162f93aef.slice/crio-e71321c18ed0047c9b6b2e6074801517e0e145d0f91f5ee1ab359fab12af8960 WatchSource:0}: Error finding container e71321c18ed0047c9b6b2e6074801517e0e145d0f91f5ee1ab359fab12af8960: Status 404 returned error can't find the container with id e71321c18ed0047c9b6b2e6074801517e0e145d0f91f5ee1ab359fab12af8960
Sep 09 20:48:44.919330 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:44.919306    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a618664de1a81cb8ba66310981e215f.slice/crio-fd3e57015c963e98a78567bd04e5550f42701bb7a053b995f68c9507db103f9e WatchSource:0}: Error finding container fd3e57015c963e98a78567bd04e5550f42701bb7a053b995f68c9507db103f9e: Status 404 returned error can't find the container with id fd3e57015c963e98a78567bd04e5550f42701bb7a053b995f68c9507db103f9e
Sep 09 20:48:44.923518 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.923502    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:48:44.948122 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:44.948101    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:48:45.685094 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.684813    2578 apiserver.go:52] "Watching apiserver"
Sep 09 20:48:45.693466 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.693436    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:48:45.693923 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.693891    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-75wr4","openshift-image-registry/node-ca-7tx4f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal","openshift-multus/multus-additional-cni-plugins-tpmcx","openshift-multus/multus-zrh2k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m","openshift-dns/node-resolver-jfdfz","openshift-multus/network-metrics-daemon-4wr4b","openshift-network-diagnostics/network-check-target-6v8q5","openshift-network-operator/iptables-alerter-m8xkt","openshift-ovn-kubernetes/ovnkube-node-b2lxl","kube-system/global-pull-secret-syncer-2wnz7","kube-system/konnectivity-agent-d8rzb","kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"]
Sep 09 20:48:45.697011 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.696543    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jfdfz"
Sep 09 20:48:45.699169 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.698824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:48:45.699346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.699329    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7bhcb\""
Sep 09 20:48:45.699418 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.699346    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:48:45.699561 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.699539    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:48:45.701368 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.701347    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:45.701471 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:45.701425    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:48:45.701930 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.701906    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:48:45.702175 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.702157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sl4rt\""
Sep 09 20:48:45.702560 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.702540    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:48:45.709236 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.708839    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:48:45.709236 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.709231    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.712364 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.712342    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.713550 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.712979    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:48:45.713550 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.713163    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:48:45.713550 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.713176    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:48:45.713550 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.713288    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:48:45.713550 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.713187    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fhjrn\""
Sep 09 20:48:45.713857 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.713572    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:48:45.714767 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.714574    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:48:45.714767 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.714656    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qn5k6\""
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-var-lib-cni-bin\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdbt6\" (UniqueName: \"kubernetes.io/projected/d29650ed-4c0b-4a60-8128-8b7e95d1a253-kube-api-access-xdbt6\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7fb907c-a415-4afa-977e-1d971d4701ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-var-lib-kubelet\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-hostroot\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-run-multus-certs\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7fb907c-a415-4afa-977e-1d971d4701ac-cnibin\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kc2s\" (UniqueName: \"kubernetes.io/projected/d7fb907c-a415-4afa-977e-1d971d4701ac-kube-api-access-6kc2s\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-cnibin\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd38230-7de5-4519-bf5d-db7c7ea09cc2-tmp-dir\") pod \"node-resolver-jfdfz\" (UID: \"7fd38230-7de5-4519-bf5d-db7c7ea09cc2\") " pod="openshift-dns/node-resolver-jfdfz"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwcv\" (UniqueName: \"kubernetes.io/projected/7fd38230-7de5-4519-bf5d-db7c7ea09cc2-kube-api-access-gmwcv\") pod \"node-resolver-jfdfz\" (UID: \"7fd38230-7de5-4519-bf5d-db7c7ea09cc2\") " pod="openshift-dns/node-resolver-jfdfz"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7fb907c-a415-4afa-977e-1d971d4701ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-socket-dir-parent\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.715982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-run-netns\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.716098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-conf-dir\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-etc-kubernetes\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fd38230-7de5-4519-bf5d-db7c7ea09cc2-hosts-file\") pod \"node-resolver-jfdfz\" (UID: \"7fd38230-7de5-4519-bf5d-db7c7ea09cc2\") " pod="openshift-dns/node-resolver-jfdfz"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0282f2d8-4948-4297-94a6-9da942c5a7be-agent-certs\") pod \"konnectivity-agent-d8rzb\" (UID: \"0282f2d8-4948-4297-94a6-9da942c5a7be\") " pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7fb907c-a415-4afa-977e-1d971d4701ac-system-cni-dir\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-cni-dir\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-daemon-config\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsv4c\" (UniqueName: \"kubernetes.io/projected/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-kube-api-access-dsv4c\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-var-lib-cni-multus\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7fb907c-a415-4afa-977e-1d971d4701ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-run-k8s-cni-cncf-io\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0282f2d8-4948-4297-94a6-9da942c5a7be-konnectivity-ca\") pod \"konnectivity-agent-d8rzb\" (UID: \"0282f2d8-4948-4297-94a6-9da942c5a7be\") " pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7fb907c-a415-4afa-977e-1d971d4701ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-system-cni-dir\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-cni-binary-copy\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.717151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.716771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.719749 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.719727    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:48:45.720011 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.719992    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:48:45.720220 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.720203    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:48:45.720471 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.720449    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6jhxf\""
Sep 09 20:48:45.723356 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.722996    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:45.723356 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:45.723070    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:48:45.723356 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.723224    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:48:45.724761 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.724702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.726864 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.726829    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:48:45.727285 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.727265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7tx4f"
Sep 09 20:48:45.727913 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.727895    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w2f6z\""
Sep 09 20:48:45.728304 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.728137    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:48:45.730975 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.730104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m8xkt"
Sep 09 20:48:45.730975 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.730799    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:48:45.730975 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.730806    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zbbcc\""
Sep 09 20:48:45.731175 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.731041    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:48:45.731232 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.731191    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:48:45.732398 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.732368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.737731 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.734685    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:48:45.737731 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.735725    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:48:45.737731 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.735974    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:48:45.738254 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.738227    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:48:45.738648 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.738413    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:48:45.738648 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.738427    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:48:45.738648 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.738439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:45.738648 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.738534    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:48:45.738648 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:45.738529    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:48:45.739011 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.738922    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:48:45.739087 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.738860    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rjpjf\""
Sep 09 20:48:45.739481 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.739449    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xzfr2\""
Sep 09 20:48:45.764605 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.763695    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:48:45.764605 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.763955    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:48:45.764605 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.764163    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s6b9t"
Sep 09 20:48:45.764605 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.764441    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s6b9t"
Sep 09 20:48:45.804877 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.804849    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:48:45.816960 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.816929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-kubernetes\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.817084 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.816969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.817084 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.816998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41281e59-37b7-4890-bfe0-7cdf49a4a085-env-overrides\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.817084 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.817084 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73-iptables-alerter-script\") pod \"iptables-alerter-m8xkt\" (UID: \"9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73\") " pod="openshift-network-operator/iptables-alerter-m8xkt"
Sep 09 20:48:45.817327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdbt6\" (UniqueName: \"kubernetes.io/projected/d29650ed-4c0b-4a60-8128-8b7e95d1a253-kube-api-access-xdbt6\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:45.817327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7fb907c-a415-4afa-977e-1d971d4701ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.817327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-run-multus-certs\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.817327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41281e59-37b7-4890-bfe0-7cdf49a4a085-ovnkube-config\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.817327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-device-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.817327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9v6r\" (UniqueName: \"kubernetes.io/projected/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-kube-api-access-l9v6r\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.817327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:45.817327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7fb907c-a415-4afa-977e-1d971d4701ac-cnibin\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kc2s\" (UniqueName: \"kubernetes.io/projected/d7fb907c-a415-4afa-977e-1d971d4701ac-kube-api-access-6kc2s\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd38230-7de5-4519-bf5d-db7c7ea09cc2-tmp-dir\") pod \"node-resolver-jfdfz\" (UID: \"7fd38230-7de5-4519-bf5d-db7c7ea09cc2\") " pod="openshift-dns/node-resolver-jfdfz"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwcv\" (UniqueName: \"kubernetes.io/projected/7fd38230-7de5-4519-bf5d-db7c7ea09cc2-kube-api-access-gmwcv\") pod \"node-resolver-jfdfz\" (UID: \"7fd38230-7de5-4519-bf5d-db7c7ea09cc2\") " pod="openshift-dns/node-resolver-jfdfz"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-var-lib-openvswitch\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pb64\" (UniqueName: \"kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64\") pod \"network-check-target-6v8q5\" (UID: \"c8b1dc5c-a703-449e-9530-0b1f9c503956\") " pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7fb907c-a415-4afa-977e-1d971d4701ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-run-netns\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-conf-dir\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-etc-kubernetes\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-host\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2pkh\" (UniqueName: \"kubernetes.io/projected/41281e59-37b7-4890-bfe0-7cdf49a4a085-kube-api-access-s2pkh\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-socket-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.817705 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d44783b3-1045-4e33-83de-7380e2678e62-dbus\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:45.818172 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7fb907c-a415-4afa-977e-1d971d4701ac-system-cni-dir\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.818172 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-etc-kubernetes\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.818172 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-cni-dir\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.818172 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7fb907c-a415-4afa-977e-1d971d4701ac-cnibin\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.818172 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817851    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7fb907c-a415-4afa-977e-1d971d4701ac-system-cni-dir\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.818172 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817901    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-conf-dir\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.818172 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.817944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-run-netns\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.818172 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:45.818038    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:45.818172 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:45.818122    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs podName:d29650ed-4c0b-4a60-8128-8b7e95d1a253 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:46.318098181 +0000 UTC m=+2.972878346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs") pod "network-metrics-daemon-4wr4b" (UID: "d29650ed-4c0b-4a60-8128-8b7e95d1a253") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:45.818601 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-cni-dir\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.818601 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-run-multus-certs\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.818601 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-daemon-config\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.818601 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-sysctl-d\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.818601 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7fb907c-a415-4afa-977e-1d971d4701ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.818601 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-kubelet\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.818601 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-log-socket\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.818601 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41281e59-37b7-4890-bfe0-7cdf49a4a085-ovn-node-metrics-cert\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.818601 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:45.818601 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818611    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd38230-7de5-4519-bf5d-db7c7ea09cc2-tmp-dir\") pod \"node-resolver-jfdfz\" (UID: \"7fd38230-7de5-4519-bf5d-db7c7ea09cc2\") " pod="openshift-dns/node-resolver-jfdfz"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-run-k8s-cni-cncf-io\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2xx\" (UniqueName: \"kubernetes.io/projected/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-kube-api-access-6c2xx\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73-host-slash\") pod \"iptables-alerter-m8xkt\" (UID: \"9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73\") " pod="openshift-network-operator/iptables-alerter-m8xkt"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-sys-fs\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-run-k8s-cni-cncf-io\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d44783b3-1045-4e33-83de-7380e2678e62-kubelet-config\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0282f2d8-4948-4297-94a6-9da942c5a7be-konnectivity-ca\") pod \"konnectivity-agent-d8rzb\" (UID: \"0282f2d8-4948-4297-94a6-9da942c5a7be\") " pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7fb907c-a415-4afa-977e-1d971d4701ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-systemd-units\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-cni-bin\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-daemon-config\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.818977 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-cni-netd\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.818990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41281e59-37b7-4890-bfe0-7cdf49a4a085-ovnkube-script-lib\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-cni-binary-copy\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw7kc\" (UniqueName: \"kubernetes.io/projected/9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73-kube-api-access-kw7kc\") pod \"iptables-alerter-m8xkt\" (UID: \"9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73\") " pod="openshift-network-operator/iptables-alerter-m8xkt"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-run-netns\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-etc-openvswitch\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-node-log\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-var-lib-kubelet\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-hostroot\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-sysconfig\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-run\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-run-ovn\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-cnibin\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-sys\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819343    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0282f2d8-4948-4297-94a6-9da942c5a7be-konnectivity-ca\") pod \"konnectivity-agent-d8rzb\" (UID: \"0282f2d8-4948-4297-94a6-9da942c5a7be\") " pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-lib-modules\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-tmp\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.819609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-slash\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-socket-dir-parent\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fd38230-7de5-4519-bf5d-db7c7ea09cc2-hosts-file\") pod \"node-resolver-jfdfz\" (UID: \"7fd38230-7de5-4519-bf5d-db7c7ea09cc2\") " pod="openshift-dns/node-resolver-jfdfz"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-modprobe-d\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0282f2d8-4948-4297-94a6-9da942c5a7be-agent-certs\") pod \"konnectivity-agent-d8rzb\" (UID: \"0282f2d8-4948-4297-94a6-9da942c5a7be\") " pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsv4c\" (UniqueName: \"kubernetes.io/projected/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-kube-api-access-dsv4c\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-cni-binary-copy\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-var-lib-kubelet\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-registration-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ca402f4-5c17-48e7-bafd-2fe460254bcf-serviceca\") pod \"node-ca-7tx4f\" (UID: \"5ca402f4-5c17-48e7-bafd-2fe460254bcf\") " pod="openshift-image-registry/node-ca-7tx4f"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-var-lib-kubelet\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-hostroot\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-var-lib-cni-multus\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-multus-socket-dir-parent\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-run-systemd\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819826    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-var-lib-cni-multus\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.820214 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkqcn\" (UniqueName: \"kubernetes.io/projected/5ca402f4-5c17-48e7-bafd-2fe460254bcf-kube-api-access-tkqcn\") pod \"node-ca-7tx4f\" (UID: \"5ca402f4-5c17-48e7-bafd-2fe460254bcf\") " pod="openshift-image-registry/node-ca-7tx4f"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-cnibin\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7fb907c-a415-4afa-977e-1d971d4701ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-systemd\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-run-openvswitch\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.819993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ca402f4-5c17-48e7-bafd-2fe460254bcf-host\") pod \"node-ca-7tx4f\" (UID: \"5ca402f4-5c17-48e7-bafd-2fe460254bcf\") " pod="openshift-image-registry/node-ca-7tx4f"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.820022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fd38230-7de5-4519-bf5d-db7c7ea09cc2-hosts-file\") pod \"node-resolver-jfdfz\" (UID: \"7fd38230-7de5-4519-bf5d-db7c7ea09cc2\") " pod="openshift-dns/node-resolver-jfdfz"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.820035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-sysctl-conf\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.820069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-tuned\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.820092    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.820102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7fb907c-a415-4afa-977e-1d971d4701ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.820141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-system-cni-dir\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.820197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-var-lib-cni-bin\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.820252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-system-cni-dir\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.820280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-host-var-lib-cni-bin\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.820327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7fb907c-a415-4afa-977e-1d971d4701ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.821029 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.820621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7fb907c-a415-4afa-977e-1d971d4701ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.827727 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.826646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0282f2d8-4948-4297-94a6-9da942c5a7be-agent-certs\") pod \"konnectivity-agent-d8rzb\" (UID: \"0282f2d8-4948-4297-94a6-9da942c5a7be\") " pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:48:45.827727 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.827554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdbt6\" (UniqueName: \"kubernetes.io/projected/d29650ed-4c0b-4a60-8128-8b7e95d1a253-kube-api-access-xdbt6\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:45.829712 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.829669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsv4c\" (UniqueName: \"kubernetes.io/projected/f08d5bb8-aea8-4619-a909-dd04f6d36a7a-kube-api-access-dsv4c\") pod \"multus-zrh2k\" (UID: \"f08d5bb8-aea8-4619-a909-dd04f6d36a7a\") " pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:45.830080 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.830059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwcv\" (UniqueName: \"kubernetes.io/projected/7fd38230-7de5-4519-bf5d-db7c7ea09cc2-kube-api-access-gmwcv\") pod \"node-resolver-jfdfz\" (UID: \"7fd38230-7de5-4519-bf5d-db7c7ea09cc2\") " pod="openshift-dns/node-resolver-jfdfz"
Sep 09 20:48:45.831997 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.831973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kc2s\" (UniqueName: \"kubernetes.io/projected/d7fb907c-a415-4afa-977e-1d971d4701ac-kube-api-access-6kc2s\") pod \"multus-additional-cni-plugins-tpmcx\" (UID: \"d7fb907c-a415-4afa-977e-1d971d4701ac\") " pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:45.839491 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.839444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal" event={"ID":"1a618664de1a81cb8ba66310981e215f","Type":"ContainerStarted","Data":"fd3e57015c963e98a78567bd04e5550f42701bb7a053b995f68c9507db103f9e"}
Sep 09 20:48:45.841177 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.841151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal" event={"ID":"97370ad8a194f963a134d4e162f93aef","Type":"ContainerStarted","Data":"e71321c18ed0047c9b6b2e6074801517e0e145d0f91f5ee1ab359fab12af8960"}
Sep 09 20:48:45.920520 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2xx\" (UniqueName: \"kubernetes.io/projected/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-kube-api-access-6c2xx\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.920667 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73-host-slash\") pod \"iptables-alerter-m8xkt\" (UID: \"9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73\") " pod="openshift-network-operator/iptables-alerter-m8xkt"
Sep 09 20:48:45.920667 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-sys-fs\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.920667 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d44783b3-1045-4e33-83de-7380e2678e62-kubelet-config\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:45.920667 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d44783b3-1045-4e33-83de-7380e2678e62-kubelet-config\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:45.920957 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73-host-slash\") pod \"iptables-alerter-m8xkt\" (UID: \"9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73\") " pod="openshift-network-operator/iptables-alerter-m8xkt"
Sep 09 20:48:45.920957 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-sys-fs\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.920957 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-systemd-units\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.920957 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-cni-bin\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.920957 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-cni-netd\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.920957 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41281e59-37b7-4890-bfe0-7cdf49a4a085-ovnkube-script-lib\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.920957 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-systemd-units\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.920957 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw7kc\" (UniqueName: \"kubernetes.io/projected/9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73-kube-api-access-kw7kc\") pod \"iptables-alerter-m8xkt\" (UID: \"9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73\") " pod="openshift-network-operator/iptables-alerter-m8xkt"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-run-netns\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.920990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-etc-openvswitch\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-node-log\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-sysconfig\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-run\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-run-ovn\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-sys\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-lib-modules\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-tmp\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-cni-bin\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-run-netns\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-etc-openvswitch\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.921346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-node-log\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-sysconfig\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-run\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-run-ovn\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-cni-netd\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-sys\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-lib-modules\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-slash\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-modprobe-d\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-var-lib-kubelet\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-registration-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.921990 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.921987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ca402f4-5c17-48e7-bafd-2fe460254bcf-serviceca\") pod \"node-ca-7tx4f\" (UID: \"5ca402f4-5c17-48e7-bafd-2fe460254bcf\") " pod="openshift-image-registry/node-ca-7tx4f"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-run-systemd\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkqcn\" (UniqueName: \"kubernetes.io/projected/5ca402f4-5c17-48e7-bafd-2fe460254bcf-kube-api-access-tkqcn\") pod \"node-ca-7tx4f\" (UID: \"5ca402f4-5c17-48e7-bafd-2fe460254bcf\") " pod="openshift-image-registry/node-ca-7tx4f"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-systemd\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-run-openvswitch\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ca402f4-5c17-48e7-bafd-2fe460254bcf-host\") pod \"node-ca-7tx4f\" (UID: \"5ca402f4-5c17-48e7-bafd-2fe460254bcf\") " pod="openshift-image-registry/node-ca-7tx4f"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-sysctl-conf\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-tuned\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-kubernetes\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41281e59-37b7-4890-bfe0-7cdf49a4a085-ovnkube-script-lib\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922256    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41281e59-37b7-4890-bfe0-7cdf49a4a085-env-overrides\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73-iptables-alerter-script\") pod \"iptables-alerter-m8xkt\" (UID: \"9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73\") " pod="openshift-network-operator/iptables-alerter-m8xkt"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41281e59-37b7-4890-bfe0-7cdf49a4a085-ovnkube-config\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-device-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9v6r\" (UniqueName: \"kubernetes.io/projected/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-kube-api-access-l9v6r\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.922574 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-run-openvswitch\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-var-lib-openvswitch\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-var-lib-openvswitch\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pb64\" (UniqueName: \"kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64\") pod \"network-check-target-6v8q5\" (UID: \"c8b1dc5c-a703-449e-9530-0b1f9c503956\") " pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ca402f4-5c17-48e7-bafd-2fe460254bcf-host\") pod \"node-ca-7tx4f\" (UID: \"5ca402f4-5c17-48e7-bafd-2fe460254bcf\") " pod="openshift-image-registry/node-ca-7tx4f"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-host\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922538    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2pkh\" (UniqueName: \"kubernetes.io/projected/41281e59-37b7-4890-bfe0-7cdf49a4a085-kube-api-access-s2pkh\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-modprobe-d\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-socket-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d44783b3-1045-4e33-83de-7380e2678e62-dbus\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-slash\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-sysctl-conf\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-sysctl-d\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-kubelet\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-log-socket\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41281e59-37b7-4890-bfe0-7cdf49a4a085-ovn-node-metrics-cert\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.923507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:45.924994 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.924445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.924994 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.924553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-kubernetes\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.924994 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.924612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.924994 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.924744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-run-systemd\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.924994 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.924953    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-tmp\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.925346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.925227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-device-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.925346 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.925287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-systemd\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.925466 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.925356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.925466 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.925413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-host\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.925466 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-registration-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.925803 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.925777    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41281e59-37b7-4890-bfe0-7cdf49a4a085-env-overrides\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.926017 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.925999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d44783b3-1045-4e33-83de-7380e2678e62-dbus\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:45.926147 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.926128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-sysctl-d\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.926238 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.926188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-kubelet\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.926238 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.926206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41281e59-37b7-4890-bfe0-7cdf49a4a085-ovnkube-config\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.926238 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.926231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-socket-dir\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:45.926413 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.926300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-log-socket\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.926413 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.926311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73-iptables-alerter-script\") pod \"iptables-alerter-m8xkt\" (UID: \"9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73\") " pod="openshift-network-operator/iptables-alerter-m8xkt"
Sep 09 20:48:45.926413 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.926358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-var-lib-kubelet\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.926566 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:45.926445    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:45.926566 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:45.926509    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret podName:d44783b3-1045-4e33-83de-7380e2678e62 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:46.426491229 +0000 UTC m=+3.081271386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret") pod "global-pull-secret-syncer-2wnz7" (UID: "d44783b3-1045-4e33-83de-7380e2678e62") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:45.926792 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.922790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41281e59-37b7-4890-bfe0-7cdf49a4a085-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.927141 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.927117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-etc-tuned\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.927229 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.927150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ca402f4-5c17-48e7-bafd-2fe460254bcf-serviceca\") pod \"node-ca-7tx4f\" (UID: \"5ca402f4-5c17-48e7-bafd-2fe460254bcf\") " pod="openshift-image-registry/node-ca-7tx4f"
Sep 09 20:48:45.930881 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:45.930834    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:48:45.930881 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:45.930858    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:48:45.930881 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:45.930874    2578 projected.go:196] Error preparing data for projected volume kube-api-access-2pb64 for pod openshift-network-diagnostics/network-check-target-6v8q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:45.931103 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:45.930953    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64 podName:c8b1dc5c-a703-449e-9530-0b1f9c503956 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:46.430933063 +0000 UTC m=+3.085713217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2pb64" (UniqueName: "kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64") pod "network-check-target-6v8q5" (UID: "c8b1dc5c-a703-449e-9530-0b1f9c503956") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:45.931843 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.931816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw7kc\" (UniqueName: \"kubernetes.io/projected/9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73-kube-api-access-kw7kc\") pod \"iptables-alerter-m8xkt\" (UID: \"9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73\") " pod="openshift-network-operator/iptables-alerter-m8xkt"
Sep 09 20:48:45.932056 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.932031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41281e59-37b7-4890-bfe0-7cdf49a4a085-ovn-node-metrics-cert\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.933106 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.933024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2xx\" (UniqueName: \"kubernetes.io/projected/d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4-kube-api-access-6c2xx\") pod \"tuned-75wr4\" (UID: \"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4\") " pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:45.933686 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.933664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkqcn\" (UniqueName: \"kubernetes.io/projected/5ca402f4-5c17-48e7-bafd-2fe460254bcf-kube-api-access-tkqcn\") pod \"node-ca-7tx4f\" (UID: \"5ca402f4-5c17-48e7-bafd-2fe460254bcf\") " pod="openshift-image-registry/node-ca-7tx4f"
Sep 09 20:48:45.933991 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.933940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2pkh\" (UniqueName: \"kubernetes.io/projected/41281e59-37b7-4890-bfe0-7cdf49a4a085-kube-api-access-s2pkh\") pod \"ovnkube-node-b2lxl\" (UID: \"41281e59-37b7-4890-bfe0-7cdf49a4a085\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:45.935374 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:45.935316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9v6r\" (UniqueName: \"kubernetes.io/projected/4a6d7bb5-043c-449b-bbca-a9f0286f03a6-kube-api-access-l9v6r\") pod \"aws-ebs-csi-driver-node-kbn9m\" (UID: \"4a6d7bb5-043c-449b-bbca-a9f0286f03a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:46.014102 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.014065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jfdfz"
Sep 09 20:48:46.024450 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.024418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:48:46.037552 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.037526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tpmcx"
Sep 09 20:48:46.045442 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.045419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zrh2k"
Sep 09 20:48:46.054319 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.054295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m"
Sep 09 20:48:46.062293 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.062272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-75wr4"
Sep 09 20:48:46.072360 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.072340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7tx4f"
Sep 09 20:48:46.082215 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.082194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m8xkt"
Sep 09 20:48:46.090015 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.089995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:48:46.330278 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.330177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:46.330446 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:46.330309    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:46.330446 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:46.330416    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs podName:d29650ed-4c0b-4a60-8128-8b7e95d1a253 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:47.330392874 +0000 UTC m=+3.985173029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs") pod "network-metrics-daemon-4wr4b" (UID: "d29650ed-4c0b-4a60-8128-8b7e95d1a253") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:46.431079 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.431042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pb64\" (UniqueName: \"kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64\") pod \"network-check-target-6v8q5\" (UID: \"c8b1dc5c-a703-449e-9530-0b1f9c503956\") " pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:46.431079 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.431094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:46.431373 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:46.431228    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:46.431373 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:46.431246    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:48:46.431373 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:46.431267    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:48:46.431373 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:46.431279    2578 projected.go:196] Error preparing data for projected volume kube-api-access-2pb64 for pod openshift-network-diagnostics/network-check-target-6v8q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:46.431373 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:46.431302    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret podName:d44783b3-1045-4e33-83de-7380e2678e62 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:47.431282186 +0000 UTC m=+4.086062323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret") pod "global-pull-secret-syncer-2wnz7" (UID: "d44783b3-1045-4e33-83de-7380e2678e62") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:46.431373 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:46.431337    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64 podName:c8b1dc5c-a703-449e-9530-0b1f9c503956 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:47.431318117 +0000 UTC m=+4.086098279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pb64" (UniqueName: "kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64") pod "network-check-target-6v8q5" (UID: "c8b1dc5c-a703-449e-9530-0b1f9c503956") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:46.769203 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.769094    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:43:46 +0000 UTC" deadline="2028-03-06 11:39:53.780575407 +0000 UTC"
Sep 09 20:48:46.769203 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:46.769140    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13046h51m7.011439519s"
Sep 09 20:48:47.035032 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:47.034961    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6d7bb5_043c_449b_bbca_a9f0286f03a6.slice/crio-760c566fda5946e651719264454d0e741c72dbf77747400c0afd31607f8a5ed7 WatchSource:0}: Error finding container 760c566fda5946e651719264454d0e741c72dbf77747400c0afd31607f8a5ed7: Status 404 returned error can't find the container with id 760c566fda5946e651719264454d0e741c72dbf77747400c0afd31607f8a5ed7
Sep 09 20:48:47.036157 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:47.036135    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0282f2d8_4948_4297_94a6_9da942c5a7be.slice/crio-f9567a4df8b6986e8e57bc8b481605ab5c440ee050ad5edbdb93a487b82dc42a WatchSource:0}: Error finding container f9567a4df8b6986e8e57bc8b481605ab5c440ee050ad5edbdb93a487b82dc42a: Status 404 returned error can't find the container with id f9567a4df8b6986e8e57bc8b481605ab5c440ee050ad5edbdb93a487b82dc42a
Sep 09 20:48:47.037542 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:47.037425    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca402f4_5c17_48e7_bafd_2fe460254bcf.slice/crio-f4946abc7b4ef1dd58e97f7461bec77ac69cc1f98e9a4c7d4d8e20ad227c2646 WatchSource:0}: Error finding container f4946abc7b4ef1dd58e97f7461bec77ac69cc1f98e9a4c7d4d8e20ad227c2646: Status 404 returned error can't find the container with id f4946abc7b4ef1dd58e97f7461bec77ac69cc1f98e9a4c7d4d8e20ad227c2646
Sep 09 20:48:47.037964 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:47.037936    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41281e59_37b7_4890_bfe0_7cdf49a4a085.slice/crio-d600d07556e8da60377608d3a9df0555bd42c7150547b4ce54bd8fbc3abc243c WatchSource:0}: Error finding container d600d07556e8da60377608d3a9df0555bd42c7150547b4ce54bd8fbc3abc243c: Status 404 returned error can't find the container with id d600d07556e8da60377608d3a9df0555bd42c7150547b4ce54bd8fbc3abc243c
Sep 09 20:48:47.061805 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:47.061781    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd653ffb5_c7b1_4b99_8f9b_003d2ba4eec4.slice/crio-63db1a57d2908a241f93f8ab50bcb3a890aa000b42a5606378b36e9b2e37b3c7 WatchSource:0}: Error finding container 63db1a57d2908a241f93f8ab50bcb3a890aa000b42a5606378b36e9b2e37b3c7: Status 404 returned error can't find the container with id 63db1a57d2908a241f93f8ab50bcb3a890aa000b42a5606378b36e9b2e37b3c7
Sep 09 20:48:47.062840 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:47.062817    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd38230_7de5_4519_bf5d_db7c7ea09cc2.slice/crio-7748e377c27b022751170f4f72924e0f92cda052e0776cbfb7dd7162a6eea1c5 WatchSource:0}: Error finding container 7748e377c27b022751170f4f72924e0f92cda052e0776cbfb7dd7162a6eea1c5: Status 404 returned error can't find the container with id 7748e377c27b022751170f4f72924e0f92cda052e0776cbfb7dd7162a6eea1c5
Sep 09 20:48:47.063859 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:47.063839    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de4c5f7_4a99_4dea_a5a3_eaf97b3caf73.slice/crio-19eb4bb404f18b1c87871cb7ed9c61bf6672e10572beb5d6feac061417df6013 WatchSource:0}: Error finding container 19eb4bb404f18b1c87871cb7ed9c61bf6672e10572beb5d6feac061417df6013: Status 404 returned error can't find the container with id 19eb4bb404f18b1c87871cb7ed9c61bf6672e10572beb5d6feac061417df6013
Sep 09 20:48:47.066056 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:47.065664    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08d5bb8_aea8_4619_a909_dd04f6d36a7a.slice/crio-2aa67be892088edb101ef59349f68b89e64ad1a5a118f3c59ee2a6478fb24df6 WatchSource:0}: Error finding container 2aa67be892088edb101ef59349f68b89e64ad1a5a118f3c59ee2a6478fb24df6: Status 404 returned error can't find the container with id 2aa67be892088edb101ef59349f68b89e64ad1a5a118f3c59ee2a6478fb24df6
Sep 09 20:48:47.066056 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:48:47.066029    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fb907c_a415_4afa_977e_1d971d4701ac.slice/crio-32d959e554b119c624873e5c54e5022510ea30bcd1fc08e0479f496a7774e535 WatchSource:0}: Error finding container 32d959e554b119c624873e5c54e5022510ea30bcd1fc08e0479f496a7774e535: Status 404 returned error can't find the container with id 32d959e554b119c624873e5c54e5022510ea30bcd1fc08e0479f496a7774e535
Sep 09 20:48:47.337730 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.337700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:47.337830 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:47.337811    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:47.337877 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:47.337858    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs podName:d29650ed-4c0b-4a60-8128-8b7e95d1a253 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:49.337843781 +0000 UTC m=+5.992623919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs") pod "network-metrics-daemon-4wr4b" (UID: "d29650ed-4c0b-4a60-8128-8b7e95d1a253") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:47.438109 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.438068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:47.438293 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.438165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pb64\" (UniqueName: \"kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64\") pod \"network-check-target-6v8q5\" (UID: \"c8b1dc5c-a703-449e-9530-0b1f9c503956\") " pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:47.438293 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:47.438266    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:47.438395 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:47.438305    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:48:47.438395 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:47.438333    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:48:47.438395 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:47.438351    2578 projected.go:196] Error preparing data for projected volume kube-api-access-2pb64 for pod openshift-network-diagnostics/network-check-target-6v8q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:47.438395 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:47.438336    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret podName:d44783b3-1045-4e33-83de-7380e2678e62 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:49.438313993 +0000 UTC m=+6.093094132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret") pod "global-pull-secret-syncer-2wnz7" (UID: "d44783b3-1045-4e33-83de-7380e2678e62") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:47.438530 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:47.438417    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64 podName:c8b1dc5c-a703-449e-9530-0b1f9c503956 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:49.438399109 +0000 UTC m=+6.093179257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pb64" (UniqueName: "kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64") pod "network-check-target-6v8q5" (UID: "c8b1dc5c-a703-449e-9530-0b1f9c503956") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:47.467328 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.467302    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:48:47.770469 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.770372    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:43:46 +0000 UTC" deadline="2028-02-23 21:45:51.354464277 +0000 UTC"
Sep 09 20:48:47.770469 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.770415    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12768h57m3.584054045s"
Sep 09 20:48:47.838478 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.838441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:47.838650 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:47.838587    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:48:47.839857 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.838981    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:47.839857 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:47.839079    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:48:47.839857 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.839134    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:47.839857 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:47.839202    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:48:47.869368 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.869333    2578 generic.go:356] "Generic (PLEG): container finished" podID="97370ad8a194f963a134d4e162f93aef" containerID="8db389fd34e5519a6338cc7e539ea198f93d5fd56c3d9b4e3e6fa11a25d5e6e3" exitCode=0
Sep 09 20:48:47.869529 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.869440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal" event={"ID":"97370ad8a194f963a134d4e162f93aef","Type":"ContainerDied","Data":"8db389fd34e5519a6338cc7e539ea198f93d5fd56c3d9b4e3e6fa11a25d5e6e3"}
Sep 09 20:48:47.887572 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.885895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m8xkt" event={"ID":"9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73","Type":"ContainerStarted","Data":"19eb4bb404f18b1c87871cb7ed9c61bf6672e10572beb5d6feac061417df6013"}
Sep 09 20:48:47.898921 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.896344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jfdfz" event={"ID":"7fd38230-7de5-4519-bf5d-db7c7ea09cc2","Type":"ContainerStarted","Data":"7748e377c27b022751170f4f72924e0f92cda052e0776cbfb7dd7162a6eea1c5"}
Sep 09 20:48:47.909017 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.908980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-75wr4" event={"ID":"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4","Type":"ContainerStarted","Data":"63db1a57d2908a241f93f8ab50bcb3a890aa000b42a5606378b36e9b2e37b3c7"}
Sep 09 20:48:47.917922 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.917584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl" event={"ID":"41281e59-37b7-4890-bfe0-7cdf49a4a085","Type":"ContainerStarted","Data":"d600d07556e8da60377608d3a9df0555bd42c7150547b4ce54bd8fbc3abc243c"}
Sep 09 20:48:47.924898 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.923495    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7tx4f" event={"ID":"5ca402f4-5c17-48e7-bafd-2fe460254bcf","Type":"ContainerStarted","Data":"f4946abc7b4ef1dd58e97f7461bec77ac69cc1f98e9a4c7d4d8e20ad227c2646"}
Sep 09 20:48:47.942660 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.942275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal" event={"ID":"1a618664de1a81cb8ba66310981e215f","Type":"ContainerStarted","Data":"d207c45fcabfc35c97ab5613242ac4787cf41ec1dc2c4c36ec6d2dabbe1aad43"}
Sep 09 20:48:47.946818 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.946732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpmcx" event={"ID":"d7fb907c-a415-4afa-977e-1d971d4701ac","Type":"ContainerStarted","Data":"32d959e554b119c624873e5c54e5022510ea30bcd1fc08e0479f496a7774e535"}
Sep 09 20:48:47.955931 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.955867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zrh2k" event={"ID":"f08d5bb8-aea8-4619-a909-dd04f6d36a7a","Type":"ContainerStarted","Data":"2aa67be892088edb101ef59349f68b89e64ad1a5a118f3c59ee2a6478fb24df6"}
Sep 09 20:48:47.958475 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.957567    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal" podStartSLOduration=2.9575515279999998 podStartE2EDuration="2.957551528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:47.957358028 +0000 UTC m=+4.612138196" watchObservedRunningTime="2026-09-09 20:48:47.957551528 +0000 UTC m=+4.612331692"
Sep 09 20:48:47.960612 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.960578    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d8rzb" event={"ID":"0282f2d8-4948-4297-94a6-9da942c5a7be","Type":"ContainerStarted","Data":"f9567a4df8b6986e8e57bc8b481605ab5c440ee050ad5edbdb93a487b82dc42a"}
Sep 09 20:48:47.967281 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:47.967256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m" event={"ID":"4a6d7bb5-043c-449b-bbca-a9f0286f03a6","Type":"ContainerStarted","Data":"760c566fda5946e651719264454d0e741c72dbf77747400c0afd31607f8a5ed7"}
Sep 09 20:48:48.992926 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:48.992274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal" event={"ID":"97370ad8a194f963a134d4e162f93aef","Type":"ContainerStarted","Data":"6e86affca78c88fc956f1b77b3cf05f0478eed16ca521df8f6a107665d74bea5"}
Sep 09 20:48:49.008338 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:49.008284    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal" podStartSLOduration=4.00826017 podStartE2EDuration="4.00826017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:49.008046854 +0000 UTC m=+5.662827017" watchObservedRunningTime="2026-09-09 20:48:49.00826017 +0000 UTC m=+5.663040333"
Sep 09 20:48:49.366640 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:49.366545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:49.366862 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:49.366755    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:49.366862 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:49.366830    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs podName:d29650ed-4c0b-4a60-8128-8b7e95d1a253 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:53.366809344 +0000 UTC m=+10.021589491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs") pod "network-metrics-daemon-4wr4b" (UID: "d29650ed-4c0b-4a60-8128-8b7e95d1a253") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:49.467458 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:49.467415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pb64\" (UniqueName: \"kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64\") pod \"network-check-target-6v8q5\" (UID: \"c8b1dc5c-a703-449e-9530-0b1f9c503956\") " pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:49.467655 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:49.467485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:49.467655 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:49.467631    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:49.467778 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:49.467704    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret podName:d44783b3-1045-4e33-83de-7380e2678e62 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:53.467678975 +0000 UTC m=+10.122459135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret") pod "global-pull-secret-syncer-2wnz7" (UID: "d44783b3-1045-4e33-83de-7380e2678e62") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:49.468275 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:49.468129    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:48:49.468275 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:49.468154    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:48:49.468275 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:49.468168    2578 projected.go:196] Error preparing data for projected volume kube-api-access-2pb64 for pod openshift-network-diagnostics/network-check-target-6v8q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:49.468275 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:49.468214    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64 podName:c8b1dc5c-a703-449e-9530-0b1f9c503956 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:53.468199608 +0000 UTC m=+10.122979749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pb64" (UniqueName: "kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64") pod "network-check-target-6v8q5" (UID: "c8b1dc5c-a703-449e-9530-0b1f9c503956") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:49.838076 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:49.837989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:49.838265 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:49.838138    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:48:49.838607 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:49.838561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:49.838706 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:49.838691    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:48:49.838768 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:49.838757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:49.838857 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:49.838835    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:48:51.838496 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:51.838460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:51.838995 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:51.838629    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:48:51.839062 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:51.839015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:51.839131 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:51.839108    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:48:51.839188 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:51.839160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:51.839270 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:51.839236    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:48:53.403750 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:53.403485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:53.403750 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:53.403643    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:53.403750 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:53.403721    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs podName:d29650ed-4c0b-4a60-8128-8b7e95d1a253 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:01.403700811 +0000 UTC m=+18.058480952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs") pod "network-metrics-daemon-4wr4b" (UID: "d29650ed-4c0b-4a60-8128-8b7e95d1a253") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:53.504631 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:53.504573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pb64\" (UniqueName: \"kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64\") pod \"network-check-target-6v8q5\" (UID: \"c8b1dc5c-a703-449e-9530-0b1f9c503956\") " pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:53.504838 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:53.504644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:53.504838 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:53.504799    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:53.504956 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:53.504864    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret podName:d44783b3-1045-4e33-83de-7380e2678e62 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:01.504844879 +0000 UTC m=+18.159625037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret") pod "global-pull-secret-syncer-2wnz7" (UID: "d44783b3-1045-4e33-83de-7380e2678e62") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:53.505342 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:53.505319    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:48:53.505342 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:53.505344    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:48:53.505516 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:53.505358    2578 projected.go:196] Error preparing data for projected volume kube-api-access-2pb64 for pod openshift-network-diagnostics/network-check-target-6v8q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:53.505516 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:53.505409    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64 podName:c8b1dc5c-a703-449e-9530-0b1f9c503956 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:01.505392527 +0000 UTC m=+18.160172680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pb64" (UniqueName: "kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64") pod "network-check-target-6v8q5" (UID: "c8b1dc5c-a703-449e-9530-0b1f9c503956") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:53.836176 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:53.836091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:53.836379 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:53.836213    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:48:53.836985 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:53.836637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:53.836985 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:53.836739    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:48:53.836985 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:53.836845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:53.836985 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:53.836946    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:48:55.835464 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:55.835338    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:55.835895 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:55.835506    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:48:55.835895 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:55.835695    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:55.835895 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:55.835737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:55.835895 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:55.835813    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:48:55.836095 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:55.835900    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:48:57.835668 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:57.835635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:57.836087 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:57.835635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:57.836087 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:57.835761    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:48:57.836087 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:57.835789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:57.836087 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:57.835930    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:48:57.836087 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:57.836018    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:48:59.835776 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:59.835745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:48:59.836247 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:59.835745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:48:59.836247 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:59.835862    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:48:59.836247 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:48:59.835756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:48:59.836247 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:59.835943    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:48:59.836247 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:48:59.836024    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:49:01.462914 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:01.462884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:01.463293 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:01.463013    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:49:01.463293 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:01.463066    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs podName:d29650ed-4c0b-4a60-8128-8b7e95d1a253 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:17.463048866 +0000 UTC m=+34.117829018 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs") pod "network-metrics-daemon-4wr4b" (UID: "d29650ed-4c0b-4a60-8128-8b7e95d1a253") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:49:01.563781 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:01.563748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pb64\" (UniqueName: \"kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64\") pod \"network-check-target-6v8q5\" (UID: \"c8b1dc5c-a703-449e-9530-0b1f9c503956\") " pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:01.563781 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:01.563785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:01.563953 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:01.563877    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:49:01.563953 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:01.563894    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:49:01.563953 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:01.563913    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:49:01.563953 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:01.563923    2578 projected.go:196] Error preparing data for projected volume kube-api-access-2pb64 for pod openshift-network-diagnostics/network-check-target-6v8q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:49:01.563953 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:01.563924    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret podName:d44783b3-1045-4e33-83de-7380e2678e62 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:17.563911261 +0000 UTC m=+34.218691397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret") pod "global-pull-secret-syncer-2wnz7" (UID: "d44783b3-1045-4e33-83de-7380e2678e62") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:49:01.564122 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:01.563967    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64 podName:c8b1dc5c-a703-449e-9530-0b1f9c503956 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:17.563956237 +0000 UTC m=+34.218736377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pb64" (UniqueName: "kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64") pod "network-check-target-6v8q5" (UID: "c8b1dc5c-a703-449e-9530-0b1f9c503956") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:49:01.835808 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:01.835777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:01.835808 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:01.835792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:01.836042 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:01.835914    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:49:01.836042 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:01.835936    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:01.836042 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:01.836036    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:49:01.836235 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:01.836111    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:49:03.836065 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:03.836025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:03.836568 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:03.836133    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:49:03.836568 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:03.836217    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:03.836568 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:03.836338    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:49:03.836568 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:03.836382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:03.836568 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:03.836444    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:49:05.836236 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:05.835794    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:05.837167 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:05.835808    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:05.837167 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:05.835948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:05.837167 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:05.836731    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:49:05.837167 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:05.836967    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:49:05.837818 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:05.837786    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:49:06.030838 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.030730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zrh2k" event={"ID":"f08d5bb8-aea8-4619-a909-dd04f6d36a7a","Type":"ContainerStarted","Data":"06934a84f88d5479cf48bc1030aae5bdc4b1a6e2f6bc0385d6936538ae068f3a"}
Sep 09 20:49:06.032590 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.032563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d8rzb" event={"ID":"0282f2d8-4948-4297-94a6-9da942c5a7be","Type":"ContainerStarted","Data":"be4f77e21b857c187e81af3a39e25d1b47aaf62ae920ba2350ea09ef0977a1cb"}
Sep 09 20:49:06.034359 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.034326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m" event={"ID":"4a6d7bb5-043c-449b-bbca-a9f0286f03a6","Type":"ContainerStarted","Data":"c4fcdfbe66326cd607bc90e3edc9f64c08d5ebac1386af2e5347cef15f59bd2a"}
Sep 09 20:49:06.036120 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.036058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jfdfz" event={"ID":"7fd38230-7de5-4519-bf5d-db7c7ea09cc2","Type":"ContainerStarted","Data":"ed719b138b7c03cdf03e43611c5eef5ea76c303e37139e2c5cc5d4398c5ac19c"}
Sep 09 20:49:06.037663 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.037640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-75wr4" event={"ID":"d653ffb5-c7b1-4b99-8f9b-003d2ba4eec4","Type":"ContainerStarted","Data":"df45f059dbc81e90922a935d3e063cb0437f55a2f9ec2b34c79bc8a43b3cee1e"}
Sep 09 20:49:06.041200 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.041162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl" event={"ID":"41281e59-37b7-4890-bfe0-7cdf49a4a085","Type":"ContainerStarted","Data":"fd04ed6c30919e37bfc6dd3859373535d3b0aa3195623266b6856c08fa221a45"}
Sep 09 20:49:06.041295 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.041249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl" event={"ID":"41281e59-37b7-4890-bfe0-7cdf49a4a085","Type":"ContainerStarted","Data":"7520dac64c671d02ba7e580e4635b69ae5053412e9a63cef578b0e7d66e66a60"}
Sep 09 20:49:06.041352 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.041300    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl" event={"ID":"41281e59-37b7-4890-bfe0-7cdf49a4a085","Type":"ContainerStarted","Data":"24e718763b82a1d85b296782da8fffcb6ca6b0085a189b30c2ad77e2c620b7a8"}
Sep 09 20:49:06.041352 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.041347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl" event={"ID":"41281e59-37b7-4890-bfe0-7cdf49a4a085","Type":"ContainerStarted","Data":"1088f2e1ee06d9091bb4b5b59b06830f3a6d661272cc242e682f79250b00c949"}
Sep 09 20:49:06.041493 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.041392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl" event={"ID":"41281e59-37b7-4890-bfe0-7cdf49a4a085","Type":"ContainerStarted","Data":"9e3fd3f77156f9407d5c41a66c27076970c2886e0f828fecdb5d29f578007948"}
Sep 09 20:49:06.041493 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.041424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl" event={"ID":"41281e59-37b7-4890-bfe0-7cdf49a4a085","Type":"ContainerStarted","Data":"3f6a213cd05d8bde723d3cce2d2e991790c049872fcd8c43a4eeb18770bb6ead"}
Sep 09 20:49:06.042778 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.042755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7tx4f" event={"ID":"5ca402f4-5c17-48e7-bafd-2fe460254bcf","Type":"ContainerStarted","Data":"ddfe7c2e50f09f5c8da5cfe40db558ab42d024c676c13e6b5d3afaad7370b5d4"}
Sep 09 20:49:06.044322 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.044298    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7fb907c-a415-4afa-977e-1d971d4701ac" containerID="3c7d49b6dd133672bacab11b31af399a3aaaff6989a2848e6c357a28da1f388e" exitCode=0
Sep 09 20:49:06.044415 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.044333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpmcx" event={"ID":"d7fb907c-a415-4afa-977e-1d971d4701ac","Type":"ContainerDied","Data":"3c7d49b6dd133672bacab11b31af399a3aaaff6989a2848e6c357a28da1f388e"}
Sep 09 20:49:06.046770 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.046732    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zrh2k" podStartSLOduration=4.063227169 podStartE2EDuration="22.046717088s" totalImagesPullingTime="17.983489919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:06.04615798 +0000 UTC m=+22.700938138" watchObservedRunningTime="2026-09-09 20:49:06.046717088 +0000 UTC m=+22.701497247"
Sep 09 20:49:06.062023 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.061977    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-75wr4" podStartSLOduration=4.096662029 podStartE2EDuration="22.06196043s" totalImagesPullingTime="17.965298401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:06.061322704 +0000 UTC m=+22.716102861" watchObservedRunningTime="2026-09-09 20:49:06.06196043 +0000 UTC m=+22.716740591"
Sep 09 20:49:06.074463 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.074424    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7tx4f" podStartSLOduration=4.124429887 podStartE2EDuration="22.074408868s" totalImagesPullingTime="17.949978981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:06.074155005 +0000 UTC m=+22.728935163" watchObservedRunningTime="2026-09-09 20:49:06.074408868 +0000 UTC m=+22.729189028"
Sep 09 20:49:06.087434 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.087392    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jfdfz" podStartSLOduration=4.147445218 podStartE2EDuration="22.087375647s" totalImagesPullingTime="17.939930429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:06.086906551 +0000 UTC m=+22.741686708" watchObservedRunningTime="2026-09-09 20:49:06.087375647 +0000 UTC m=+22.742155808"
Sep 09 20:49:06.100861 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.100822    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d8rzb" podStartSLOduration=4.154881496 podStartE2EDuration="22.100808038s" totalImagesPullingTime="17.945926542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:06.10075794 +0000 UTC m=+22.755538098" watchObservedRunningTime="2026-09-09 20:49:06.100808038 +0000 UTC m=+22.755588197"
Sep 09 20:49:06.475613 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.475594    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:49:06.768798 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.768649    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:49:06.475609903Z","UUID":"ec61fe7d-9e3d-4e6c-a972-27f1bd84c4ab","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:49:06.770024 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.770003    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:49:06.770131 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:06.770039    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:49:07.048387 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:07.048314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m" event={"ID":"4a6d7bb5-043c-449b-bbca-a9f0286f03a6","Type":"ContainerStarted","Data":"b0431274a6d8434eec433d45970011c3d1e3af1870b711b99a50579261cf1b44"}
Sep 09 20:49:07.049781 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:07.049761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m8xkt" event={"ID":"9de4c5f7-4a99-4dea-a5a3-eaf97b3caf73","Type":"ContainerStarted","Data":"38ae88669e398c4b0db348889c01ec4b48bfee30933ff5ceb87c83a58cfb90e9"}
Sep 09 20:49:07.065009 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:07.064971    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m8xkt" podStartSLOduration=5.126243484 podStartE2EDuration="23.064957691s" totalImagesPullingTime="17.938714207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:07.064706937 +0000 UTC m=+23.719487119" watchObservedRunningTime="2026-09-09 20:49:07.064957691 +0000 UTC m=+23.719737852"
Sep 09 20:49:07.743628 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:07.743591    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:49:07.744484 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:07.744462    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:49:07.835537 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:07.835450    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:07.835686 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:07.835589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:07.835686 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:07.835599    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:49:07.835806 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:07.835701    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:49:07.835806 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:07.835773    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:07.835912 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:07.835847    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:49:08.056328 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:08.056289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl" event={"ID":"41281e59-37b7-4890-bfe0-7cdf49a4a085","Type":"ContainerStarted","Data":"a423029006d61bf60fe9f81772297086de6321a9de296af5b6e343c65cda7998"}
Sep 09 20:49:08.056741 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:08.056630    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:49:08.057009 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:08.056984    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d8rzb"
Sep 09 20:49:09.060059 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:09.060024    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m" event={"ID":"4a6d7bb5-043c-449b-bbca-a9f0286f03a6","Type":"ContainerStarted","Data":"f1ec53dfb4bc18d85145ae336f215a1e05957387887fbbc65fd3c85db37cb7e8"}
Sep 09 20:49:09.076656 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:09.076608    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbn9m" podStartSLOduration=4.377412701 podStartE2EDuration="25.076588974s" totalImagesPullingTime="20.699176273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:09.076538869 +0000 UTC m=+25.731319038" watchObservedRunningTime="2026-09-09 20:49:09.076588974 +0000 UTC m=+25.731369130"
Sep 09 20:49:09.835794 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:09.835582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:09.835980 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:09.835582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:09.836044 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:09.836021    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:49:09.836099 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:09.836073    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:09.836279 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:09.836241    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:49:09.836370 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:09.836318    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:49:11.834918 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:11.834884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:11.835711 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:11.834999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:11.835711 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:11.835009    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:49:11.835711 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:11.835108    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:49:11.835711 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:11.835154    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:11.835711 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:11.835218    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:49:12.070271 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:12.070169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl" event={"ID":"41281e59-37b7-4890-bfe0-7cdf49a4a085","Type":"ContainerStarted","Data":"b0477bbbfc2292eda4d096ccfef3f90be038cef793fadf29e96bf10be48a38a9"}
Sep 09 20:49:12.070415 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:12.070402    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:49:12.072093 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:12.072066    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7fb907c-a415-4afa-977e-1d971d4701ac" containerID="15c38e245d8e33161086cc4e9a27b2a893953598a315f5983b38471623c24821" exitCode=0
Sep 09 20:49:12.072207 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:12.072103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpmcx" event={"ID":"d7fb907c-a415-4afa-977e-1d971d4701ac","Type":"ContainerDied","Data":"15c38e245d8e33161086cc4e9a27b2a893953598a315f5983b38471623c24821"}
Sep 09 20:49:12.085968 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:12.085946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:49:12.093827 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:12.093790    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl" podStartSLOduration=10.053067699 podStartE2EDuration="28.093778408s" totalImagesPullingTime="18.040710709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:12.093469024 +0000 UTC m=+28.748249192" watchObservedRunningTime="2026-09-09 20:49:12.093778408 +0000 UTC m=+28.748558565"
Sep 09 20:49:13.015701 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:13.015447    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2wnz7"]
Sep 09 20:49:13.016194 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:13.015836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:13.016194 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:13.015944    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:49:13.016194 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:13.016150    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6v8q5"]
Sep 09 20:49:13.016323 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:13.016250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:13.016356 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:13.016320    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:49:13.017946 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:13.017915    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4wr4b"]
Sep 09 20:49:13.018050 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:13.018029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:13.018156 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:13.018134    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:49:13.074774 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:13.074712    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:49:13.074902 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:13.074776    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:49:13.089989 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:13.089965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:49:14.079134 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:14.079099    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7fb907c-a415-4afa-977e-1d971d4701ac" containerID="965b9700c99cda58eba092cc2f910d51c422c990da5a519aab08da1184bb1c65" exitCode=0
Sep 09 20:49:14.079525 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:14.079197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpmcx" event={"ID":"d7fb907c-a415-4afa-977e-1d971d4701ac","Type":"ContainerDied","Data":"965b9700c99cda58eba092cc2f910d51c422c990da5a519aab08da1184bb1c65"}
Sep 09 20:49:14.835513 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:14.835485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:14.835649 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:14.835485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:14.835649 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:14.835583    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:49:14.835649 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:14.835485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:14.835770 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:14.835651    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:49:14.835770 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:14.835715    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:49:16.085491 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:16.085321    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7fb907c-a415-4afa-977e-1d971d4701ac" containerID="2585c6993f86ff52bf5c18c411139035b13cb6dea2e601f9adb8d9f92998693e" exitCode=0
Sep 09 20:49:16.085491 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:16.085406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpmcx" event={"ID":"d7fb907c-a415-4afa-977e-1d971d4701ac","Type":"ContainerDied","Data":"2585c6993f86ff52bf5c18c411139035b13cb6dea2e601f9adb8d9f92998693e"}
Sep 09 20:49:16.835109 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:16.835078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:16.835368 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:16.835085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:16.835368 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:16.835207    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6v8q5" podUID="c8b1dc5c-a703-449e-9530-0b1f9c503956"
Sep 09 20:49:16.835368 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:16.835085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:16.835368 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:16.835291    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2wnz7" podUID="d44783b3-1045-4e33-83de-7380e2678e62"
Sep 09 20:49:16.835539 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:16.835382    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:49:17.478861 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.478835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:17.479264 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:17.478978    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:49:17.479264 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:17.479040    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs podName:d29650ed-4c0b-4a60-8128-8b7e95d1a253 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:49.479021347 +0000 UTC m=+66.133801483 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs") pod "network-metrics-daemon-4wr4b" (UID: "d29650ed-4c0b-4a60-8128-8b7e95d1a253") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:49:17.579799 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.579766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pb64\" (UniqueName: \"kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64\") pod \"network-check-target-6v8q5\" (UID: \"c8b1dc5c-a703-449e-9530-0b1f9c503956\") " pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:17.579931 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.579826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:17.579931 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:17.579886    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:49:17.579931 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:17.579904    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:49:17.579931 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:17.579917    2578 projected.go:196] Error preparing data for projected volume kube-api-access-2pb64 for pod openshift-network-diagnostics/network-check-target-6v8q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:49:17.580131 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:17.579973    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64 podName:c8b1dc5c-a703-449e-9530-0b1f9c503956 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:49.579958114 +0000 UTC m=+66.234738249 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pb64" (UniqueName: "kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64") pod "network-check-target-6v8q5" (UID: "c8b1dc5c-a703-449e-9530-0b1f9c503956") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:49:17.580131 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:17.579974    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:49:17.580131 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:17.580023    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret podName:d44783b3-1045-4e33-83de-7380e2678e62 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:49.580008761 +0000 UTC m=+66.234788901 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret") pod "global-pull-secret-syncer-2wnz7" (UID: "d44783b3-1045-4e33-83de-7380e2678e62") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:49:17.662309 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.662287    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeReady"
Sep 09 20:49:17.662436 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.662411    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:49:17.711506 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.711476    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nglhm"]
Sep 09 20:49:17.715969 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.715944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:17.717712 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.717681    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mtqlt"]
Sep 09 20:49:17.718491 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.718272    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5zq52\""
Sep 09 20:49:17.718614 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.718508    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:49:17.718614 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.718547    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:49:17.721258 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.721240    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mtqlt"
Sep 09 20:49:17.724185 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.724156    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:49:17.724271 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.724257    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:49:17.724343 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.724325    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:49:17.724565 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.724550    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b96lg\""
Sep 09 20:49:17.729340 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.729285    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nglhm"]
Sep 09 20:49:17.733762 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.733713    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mtqlt"]
Sep 09 20:49:17.882505 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.882477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c27b357-bfd5-4cc0-8da1-8245185755bf-tmp-dir\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:17.882505 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.882510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:17.882700 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.882530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6155565e-bace-404d-8d02-daea0e9e697d-cert\") pod \"ingress-canary-mtqlt\" (UID: \"6155565e-bace-404d-8d02-daea0e9e697d\") " pod="openshift-ingress-canary/ingress-canary-mtqlt"
Sep 09 20:49:17.882700 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.882605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c27b357-bfd5-4cc0-8da1-8245185755bf-config-volume\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:17.882700 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.882620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66rcv\" (UniqueName: \"kubernetes.io/projected/8c27b357-bfd5-4cc0-8da1-8245185755bf-kube-api-access-66rcv\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:17.882700 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.882659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98tvg\" (UniqueName: \"kubernetes.io/projected/6155565e-bace-404d-8d02-daea0e9e697d-kube-api-access-98tvg\") pod \"ingress-canary-mtqlt\" (UID: \"6155565e-bace-404d-8d02-daea0e9e697d\") " pod="openshift-ingress-canary/ingress-canary-mtqlt"
Sep 09 20:49:17.983891 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.983810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98tvg\" (UniqueName: \"kubernetes.io/projected/6155565e-bace-404d-8d02-daea0e9e697d-kube-api-access-98tvg\") pod \"ingress-canary-mtqlt\" (UID: \"6155565e-bace-404d-8d02-daea0e9e697d\") " pod="openshift-ingress-canary/ingress-canary-mtqlt"
Sep 09 20:49:17.983891 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.983861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c27b357-bfd5-4cc0-8da1-8245185755bf-tmp-dir\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:17.983891 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.983888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:17.984156 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.983913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6155565e-bace-404d-8d02-daea0e9e697d-cert\") pod \"ingress-canary-mtqlt\" (UID: \"6155565e-bace-404d-8d02-daea0e9e697d\") " pod="openshift-ingress-canary/ingress-canary-mtqlt"
Sep 09 20:49:17.984156 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.983983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c27b357-bfd5-4cc0-8da1-8245185755bf-config-volume\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:17.984156 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.984004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66rcv\" (UniqueName: \"kubernetes.io/projected/8c27b357-bfd5-4cc0-8da1-8245185755bf-kube-api-access-66rcv\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:17.984156 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:17.984029    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:17.984156 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:17.984095    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls podName:8c27b357-bfd5-4cc0-8da1-8245185755bf nodeName:}" failed. No retries permitted until 2026-09-09 20:49:18.48407399 +0000 UTC m=+35.138854130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls") pod "dns-default-nglhm" (UID: "8c27b357-bfd5-4cc0-8da1-8245185755bf") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:17.984437 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.984234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c27b357-bfd5-4cc0-8da1-8245185755bf-tmp-dir\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:17.984820 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.984790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c27b357-bfd5-4cc0-8da1-8245185755bf-config-volume\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:17.988618 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.988598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6155565e-bace-404d-8d02-daea0e9e697d-cert\") pod \"ingress-canary-mtqlt\" (UID: \"6155565e-bace-404d-8d02-daea0e9e697d\") " pod="openshift-ingress-canary/ingress-canary-mtqlt"
Sep 09 20:49:17.992483 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.992392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98tvg\" (UniqueName: \"kubernetes.io/projected/6155565e-bace-404d-8d02-daea0e9e697d-kube-api-access-98tvg\") pod \"ingress-canary-mtqlt\" (UID: \"6155565e-bace-404d-8d02-daea0e9e697d\") " pod="openshift-ingress-canary/ingress-canary-mtqlt"
Sep 09 20:49:17.992615 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:17.992492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66rcv\" (UniqueName: \"kubernetes.io/projected/8c27b357-bfd5-4cc0-8da1-8245185755bf-kube-api-access-66rcv\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:18.037183 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.037148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mtqlt"
Sep 09 20:49:18.332254 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.332224    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mtqlt"]
Sep 09 20:49:18.337650 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:49:18.337618    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6155565e_bace_404d_8d02_daea0e9e697d.slice/crio-d5ed9c7620013194f1e96e5cccc9b847eb1295cee8e4fbe8bae84e8e0212f510 WatchSource:0}: Error finding container d5ed9c7620013194f1e96e5cccc9b847eb1295cee8e4fbe8bae84e8e0212f510: Status 404 returned error can't find the container with id d5ed9c7620013194f1e96e5cccc9b847eb1295cee8e4fbe8bae84e8e0212f510
Sep 09 20:49:18.487727 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.487695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:18.488527 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:18.487861    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:18.488527 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:18.487929    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls podName:8c27b357-bfd5-4cc0-8da1-8245185755bf nodeName:}" failed. No retries permitted until 2026-09-09 20:49:19.487912158 +0000 UTC m=+36.142692311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls") pod "dns-default-nglhm" (UID: "8c27b357-bfd5-4cc0-8da1-8245185755bf") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:18.835726 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.835687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:18.835898 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.835689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:18.836150 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.835689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:18.838456 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.838433    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:49:18.839360 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.839341    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:49:18.839610 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.839577    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:49:18.839841 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.839804    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m7n9z\""
Sep 09 20:49:18.839924 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.839908    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v6qsr\""
Sep 09 20:49:18.840064 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:18.840049    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:49:19.095150 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:19.095058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mtqlt" event={"ID":"6155565e-bace-404d-8d02-daea0e9e697d","Type":"ContainerStarted","Data":"d5ed9c7620013194f1e96e5cccc9b847eb1295cee8e4fbe8bae84e8e0212f510"}
Sep 09 20:49:19.494314 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:19.494225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:19.494742 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:19.494398    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:19.494742 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:19.494484    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls podName:8c27b357-bfd5-4cc0-8da1-8245185755bf nodeName:}" failed. No retries permitted until 2026-09-09 20:49:21.4944629 +0000 UTC m=+38.149243039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls") pod "dns-default-nglhm" (UID: "8c27b357-bfd5-4cc0-8da1-8245185755bf") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:21.512929 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:21.512695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:21.513323 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:21.512817    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:21.513323 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:21.513054    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls podName:8c27b357-bfd5-4cc0-8da1-8245185755bf nodeName:}" failed. No retries permitted until 2026-09-09 20:49:25.513037006 +0000 UTC m=+42.167817149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls") pod "dns-default-nglhm" (UID: "8c27b357-bfd5-4cc0-8da1-8245185755bf") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:22.103692 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:22.103662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mtqlt" event={"ID":"6155565e-bace-404d-8d02-daea0e9e697d","Type":"ContainerStarted","Data":"d8d624ad9e67008f797f2eadc41f9f26c9d7a9c2a0c6852d2c10eee228693215"}
Sep 09 20:49:22.106333 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:22.106311    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7fb907c-a415-4afa-977e-1d971d4701ac" containerID="ad947105dd5225cb60e8d1733ad8a9884e94860195c0fe01643caebdacb83c62" exitCode=0
Sep 09 20:49:22.106422 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:22.106356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpmcx" event={"ID":"d7fb907c-a415-4afa-977e-1d971d4701ac","Type":"ContainerDied","Data":"ad947105dd5225cb60e8d1733ad8a9884e94860195c0fe01643caebdacb83c62"}
Sep 09 20:49:22.118381 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:22.118344    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mtqlt" podStartSLOduration=2.095469017 podStartE2EDuration="5.118330457s" totalImagesPullingTime="3.02286144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:22.118030549 +0000 UTC m=+38.772810754" watchObservedRunningTime="2026-09-09 20:49:22.118330457 +0000 UTC m=+38.773110616"
Sep 09 20:49:23.112121 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:23.112091    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7fb907c-a415-4afa-977e-1d971d4701ac" containerID="0c21d8d1a1e9da70e2a7f4377cf8a8ad3f4795f8fe97880fc2b472c4c698f455" exitCode=0
Sep 09 20:49:23.112628 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:23.112187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpmcx" event={"ID":"d7fb907c-a415-4afa-977e-1d971d4701ac","Type":"ContainerDied","Data":"0c21d8d1a1e9da70e2a7f4377cf8a8ad3f4795f8fe97880fc2b472c4c698f455"}
Sep 09 20:49:24.117539 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:24.117502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpmcx" event={"ID":"d7fb907c-a415-4afa-977e-1d971d4701ac","Type":"ContainerStarted","Data":"cc764e31c1c63bd233aba5c8b82110fdfd5220fd90094d9aea3f2945b8ddb599"}
Sep 09 20:49:24.139394 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:24.139345    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tpmcx" podStartSLOduration=9.3361239 podStartE2EDuration="40.139331199s" totalImagesPullingTime="30.803207299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:24.138382646 +0000 UTC m=+40.793162804" watchObservedRunningTime="2026-09-09 20:49:24.139331199 +0000 UTC m=+40.794111357"
Sep 09 20:49:25.539915 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:25.539857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:25.540312 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:25.540010    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:25.540312 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:25.540076    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls podName:8c27b357-bfd5-4cc0-8da1-8245185755bf nodeName:}" failed. No retries permitted until 2026-09-09 20:49:33.540059652 +0000 UTC m=+50.194839788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls") pod "dns-default-nglhm" (UID: "8c27b357-bfd5-4cc0-8da1-8245185755bf") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:33.595672 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:33.595629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:33.596153 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:33.595739    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:33.596153 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:33.595802    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls podName:8c27b357-bfd5-4cc0-8da1-8245185755bf nodeName:}" failed. No retries permitted until 2026-09-09 20:49:49.595787987 +0000 UTC m=+66.250568123 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls") pod "dns-default-nglhm" (UID: "8c27b357-bfd5-4cc0-8da1-8245185755bf") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:45.091191 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:45.091149    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2lxl"
Sep 09 20:49:49.500605 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.500556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:49:49.502902 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.502885    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:49:49.511464 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:49.511444    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:49:49.511553 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:49.511501    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs podName:d29650ed-4c0b-4a60-8128-8b7e95d1a253 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:53.511482019 +0000 UTC m=+130.166262156 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs") pod "network-metrics-daemon-4wr4b" (UID: "d29650ed-4c0b-4a60-8128-8b7e95d1a253") : secret "metrics-daemon-secret" not found
Sep 09 20:49:49.601231 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.601195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:49.601301 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.601253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:49:49.601301 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.601283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pb64\" (UniqueName: \"kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64\") pod \"network-check-target-6v8q5\" (UID: \"c8b1dc5c-a703-449e-9530-0b1f9c503956\") " pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:49.601412 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:49.601394    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:49.601463 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:49:49.601453    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls podName:8c27b357-bfd5-4cc0-8da1-8245185755bf nodeName:}" failed. No retries permitted until 2026-09-09 20:50:21.60143498 +0000 UTC m=+98.256215135 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls") pod "dns-default-nglhm" (UID: "8c27b357-bfd5-4cc0-8da1-8245185755bf") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:49.603503 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.603489    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:49:49.603552 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.603489    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:49:49.613260 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.613236    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:49:49.614781 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.614763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44783b3-1045-4e33-83de-7380e2678e62-original-pull-secret\") pod \"global-pull-secret-syncer-2wnz7\" (UID: \"d44783b3-1045-4e33-83de-7380e2678e62\") " pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:49.624746 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.624718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pb64\" (UniqueName: \"kubernetes.io/projected/c8b1dc5c-a703-449e-9530-0b1f9c503956-kube-api-access-2pb64\") pod \"network-check-target-6v8q5\" (UID: \"c8b1dc5c-a703-449e-9530-0b1f9c503956\") " pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:49.752079 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.752006    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m7n9z\""
Sep 09 20:49:49.758961 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.758944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2wnz7"
Sep 09 20:49:49.760019 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:49.760001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:50.030423 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:50.030337    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2wnz7"]
Sep 09 20:49:50.033526 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:50.033500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6v8q5"]
Sep 09 20:49:50.034453 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:49:50.034429    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44783b3_1045_4e33_83de_7380e2678e62.slice/crio-29cd97aefa0583c4ae5007fb40eb53a3e0af76829f2dac19635c246eb9799a1f WatchSource:0}: Error finding container 29cd97aefa0583c4ae5007fb40eb53a3e0af76829f2dac19635c246eb9799a1f: Status 404 returned error can't find the container with id 29cd97aefa0583c4ae5007fb40eb53a3e0af76829f2dac19635c246eb9799a1f
Sep 09 20:49:50.042000 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:49:50.041973    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b1dc5c_a703_449e_9530_0b1f9c503956.slice/crio-751d5a5ff3ef95c4c834fd2de9ca597088af758dd04a25d4cf8057773f258d3e WatchSource:0}: Error finding container 751d5a5ff3ef95c4c834fd2de9ca597088af758dd04a25d4cf8057773f258d3e: Status 404 returned error can't find the container with id 751d5a5ff3ef95c4c834fd2de9ca597088af758dd04a25d4cf8057773f258d3e
Sep 09 20:49:50.176089 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:50.176048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6v8q5" event={"ID":"c8b1dc5c-a703-449e-9530-0b1f9c503956","Type":"ContainerStarted","Data":"751d5a5ff3ef95c4c834fd2de9ca597088af758dd04a25d4cf8057773f258d3e"}
Sep 09 20:49:50.177110 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:50.177085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2wnz7" event={"ID":"d44783b3-1045-4e33-83de-7380e2678e62","Type":"ContainerStarted","Data":"29cd97aefa0583c4ae5007fb40eb53a3e0af76829f2dac19635c246eb9799a1f"}
Sep 09 20:49:55.192962 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:55.192859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6v8q5" event={"ID":"c8b1dc5c-a703-449e-9530-0b1f9c503956","Type":"ContainerStarted","Data":"8f562171e3b400150e9a3ecb2fa43b7c1d931108bc80c5aabbbd6468568a5015"}
Sep 09 20:49:55.192962 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:55.192959    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:49:55.194353 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:55.194331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2wnz7" event={"ID":"d44783b3-1045-4e33-83de-7380e2678e62","Type":"ContainerStarted","Data":"0eaa07849231a6353a9c20e84e1cc2a62b4dba9026c48730fdaf0e9c5bbfc28c"}
Sep 09 20:49:55.207041 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:55.206996    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6v8q5" podStartSLOduration=66.386347968 podStartE2EDuration="1m11.206980036s" totalImagesPullingTime="4.820632068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:55.206300159 +0000 UTC m=+71.861080331" watchObservedRunningTime="2026-09-09 20:49:55.206980036 +0000 UTC m=+71.861760195"
Sep 09 20:49:55.219558 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:49:55.219514    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2wnz7" podStartSLOduration=66.379521257 podStartE2EDuration="1m11.219499258s" totalImagesPullingTime="4.839978001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:55.218759133 +0000 UTC m=+71.873539291" watchObservedRunningTime="2026-09-09 20:49:55.219499258 +0000 UTC m=+71.874279416"
Sep 09 20:50:15.204427 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.204393    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"]
Sep 09 20:50:15.207222 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.207206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:15.209958 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.209936    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:50:15.210061 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.209959    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 20:50:15.210061 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.209967    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:50:15.210061 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.209966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:50:15.215389 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.215367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"]
Sep 09 20:50:15.230498 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.230469    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"]
Sep 09 20:50:15.232287 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.232271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.234329 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.234313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 20:50:15.234460 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.234337    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 20:50:15.234567 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.234467    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 20:50:15.234567 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.234504    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 20:50:15.244240 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.244221    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"]
Sep 09 20:50:15.274745 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.274719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-ca\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.274848 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.274746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-hub\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.274848 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.274766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a7ed4497-4610-4522-8d49-b360b8fe07ba-klusterlet-config\") pod \"klusterlet-addon-workmgr-64cf59f8d8-2pprr\" (UID: \"a7ed4497-4610-4522-8d49-b360b8fe07ba\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:15.274848 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.274782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.274848 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.274797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.274848 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.274846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7ed4497-4610-4522-8d49-b360b8fe07ba-tmp\") pod \"klusterlet-addon-workmgr-64cf59f8d8-2pprr\" (UID: \"a7ed4497-4610-4522-8d49-b360b8fe07ba\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:15.275041 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.274880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.275041 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.274902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mpv\" (UniqueName: \"kubernetes.io/projected/a7ed4497-4610-4522-8d49-b360b8fe07ba-kube-api-access-94mpv\") pod \"klusterlet-addon-workmgr-64cf59f8d8-2pprr\" (UID: \"a7ed4497-4610-4522-8d49-b360b8fe07ba\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:15.275041 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.274921    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zrd\" (UniqueName: \"kubernetes.io/projected/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-kube-api-access-r2zrd\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.376249 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.376212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zrd\" (UniqueName: \"kubernetes.io/projected/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-kube-api-access-r2zrd\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.376391 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.376299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-ca\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.376391 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.376320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-hub\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.376391 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.376336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a7ed4497-4610-4522-8d49-b360b8fe07ba-klusterlet-config\") pod \"klusterlet-addon-workmgr-64cf59f8d8-2pprr\" (UID: \"a7ed4497-4610-4522-8d49-b360b8fe07ba\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:15.376391 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.376354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.376391 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.376377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.376611 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.376403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7ed4497-4610-4522-8d49-b360b8fe07ba-tmp\") pod \"klusterlet-addon-workmgr-64cf59f8d8-2pprr\" (UID: \"a7ed4497-4610-4522-8d49-b360b8fe07ba\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:15.376611 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.376531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.376611 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.376574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mpv\" (UniqueName: \"kubernetes.io/projected/a7ed4497-4610-4522-8d49-b360b8fe07ba-kube-api-access-94mpv\") pod \"klusterlet-addon-workmgr-64cf59f8d8-2pprr\" (UID: \"a7ed4497-4610-4522-8d49-b360b8fe07ba\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:15.376828 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.376800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7ed4497-4610-4522-8d49-b360b8fe07ba-tmp\") pod \"klusterlet-addon-workmgr-64cf59f8d8-2pprr\" (UID: \"a7ed4497-4610-4522-8d49-b360b8fe07ba\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:15.377277 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.377247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.378883 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.378851    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.379009 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.378986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-ca\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.379080 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.379017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-hub\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.379359 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.379341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.379421 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.379402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a7ed4497-4610-4522-8d49-b360b8fe07ba-klusterlet-config\") pod \"klusterlet-addon-workmgr-64cf59f8d8-2pprr\" (UID: \"a7ed4497-4610-4522-8d49-b360b8fe07ba\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:15.384230 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.384212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zrd\" (UniqueName: \"kubernetes.io/projected/7b9deee8-4ded-4cb6-b2d1-d6a9f686511a-kube-api-access-r2zrd\") pod \"cluster-proxy-proxy-agent-774b59d9cb-zxcwc\" (UID: \"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.384467 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.384446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mpv\" (UniqueName: \"kubernetes.io/projected/a7ed4497-4610-4522-8d49-b360b8fe07ba-kube-api-access-94mpv\") pod \"klusterlet-addon-workmgr-64cf59f8d8-2pprr\" (UID: \"a7ed4497-4610-4522-8d49-b360b8fe07ba\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:15.516562 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.516469    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:15.551354 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.551318    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:50:15.785327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.785250    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"]
Sep 09 20:50:15.789838 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:50:15.789813    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ed4497_4610_4522_8d49_b360b8fe07ba.slice/crio-8b65da212d17e86cf2f465de77b037011dc4b0a1f42078a579a2f125f2df920f WatchSource:0}: Error finding container 8b65da212d17e86cf2f465de77b037011dc4b0a1f42078a579a2f125f2df920f: Status 404 returned error can't find the container with id 8b65da212d17e86cf2f465de77b037011dc4b0a1f42078a579a2f125f2df920f
Sep 09 20:50:15.821108 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:15.821078    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"]
Sep 09 20:50:15.823719 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:50:15.823691    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b9deee8_4ded_4cb6_b2d1_d6a9f686511a.slice/crio-042e052dd58847c8075410cabfe27e167945ad7e75b5ded44f05e442e355c376 WatchSource:0}: Error finding container 042e052dd58847c8075410cabfe27e167945ad7e75b5ded44f05e442e355c376: Status 404 returned error can't find the container with id 042e052dd58847c8075410cabfe27e167945ad7e75b5ded44f05e442e355c376
Sep 09 20:50:16.245796 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:16.245756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" event={"ID":"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a","Type":"ContainerStarted","Data":"042e052dd58847c8075410cabfe27e167945ad7e75b5ded44f05e442e355c376"}
Sep 09 20:50:16.246772 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:16.246749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr" event={"ID":"a7ed4497-4610-4522-8d49-b360b8fe07ba","Type":"ContainerStarted","Data":"8b65da212d17e86cf2f465de77b037011dc4b0a1f42078a579a2f125f2df920f"}
Sep 09 20:50:21.628575 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:21.628540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:50:21.628926 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:50:21.628654    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:50:21.628926 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:50:21.628707    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls podName:8c27b357-bfd5-4cc0-8da1-8245185755bf nodeName:}" failed. No retries permitted until 2026-09-09 20:51:25.628691228 +0000 UTC m=+162.283471371 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls") pod "dns-default-nglhm" (UID: "8c27b357-bfd5-4cc0-8da1-8245185755bf") : secret "dns-default-metrics-tls" not found
Sep 09 20:50:22.268533 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:22.268491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr" event={"ID":"a7ed4497-4610-4522-8d49-b360b8fe07ba","Type":"ContainerStarted","Data":"da8504646a0c3d4f0419ebc4818189b25214bf5d048320418c49d21b1304a149"}
Sep 09 20:50:22.268724 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:22.268660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:22.270674 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:22.270649    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:50:22.270791 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:22.270718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" event={"ID":"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a","Type":"ContainerStarted","Data":"c9ab3d9558efbf7e15c0fa77029b32c7f0df5b6a41d1ede49b95cd197872b2e3"}
Sep 09 20:50:22.270791 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:22.270748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" event={"ID":"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a","Type":"ContainerStarted","Data":"e4387e7ab52c45a89707a7a00661c18ba5ac533dd9f956e575dfebd43e389d5e"}
Sep 09 20:50:22.270791 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:22.270760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" event={"ID":"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a","Type":"ContainerStarted","Data":"98f1f2b66520d9fe63688e8421c5704f8400ff5a0557115ce2a6e71e35d2ceb5"}
Sep 09 20:50:22.284281 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:22.284237    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr" podStartSLOduration=1.666847722 podStartE2EDuration="7.284226281s" totalImagesPullingTime="5.617378559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:22.283508016 +0000 UTC m=+98.938288174" watchObservedRunningTime="2026-09-09 20:50:22.284226281 +0000 UTC m=+98.939006439"
Sep 09 20:50:22.301073 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:22.301024    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" podStartSLOduration=1.715615721 podStartE2EDuration="7.301009774s" totalImagesPullingTime="5.585394053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:22.300415028 +0000 UTC m=+98.955195185" watchObservedRunningTime="2026-09-09 20:50:22.301009774 +0000 UTC m=+98.955789916"
Sep 09 20:50:26.198764 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:26.198733    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6v8q5"
Sep 09 20:50:53.560491 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:50:53.560454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:50:53.560944 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:50:53.560563    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:50:53.560944 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:50:53.560625    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs podName:d29650ed-4c0b-4a60-8128-8b7e95d1a253 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:55.560612434 +0000 UTC m=+252.215392570 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs") pod "network-metrics-daemon-4wr4b" (UID: "d29650ed-4c0b-4a60-8128-8b7e95d1a253") : secret "metrics-daemon-secret" not found
Sep 09 20:51:20.728794 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:51:20.728754    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nglhm" podUID="8c27b357-bfd5-4cc0-8da1-8245185755bf"
Sep 09 20:51:21.436443 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:51:21.436415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nglhm"
Sep 09 20:51:21.864688 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:51:21.864649    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:51:25.691108 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:51:25.691072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:51:25.691601 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:51:25.691217    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:51:25.691601 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:51:25.691280    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls podName:8c27b357-bfd5-4cc0-8da1-8245185755bf nodeName:}" failed. No retries permitted until 2026-09-09 20:53:27.691264861 +0000 UTC m=+284.346044998 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls") pod "dns-default-nglhm" (UID: "8c27b357-bfd5-4cc0-8da1-8245185755bf") : secret "dns-default-metrics-tls" not found
Sep 09 20:51:33.837317 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:51:33.837280    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:52:21.598633 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:52:21.598601    2578 generic.go:356] "Generic (PLEG): container finished" podID="a7ed4497-4610-4522-8d49-b360b8fe07ba" containerID="da8504646a0c3d4f0419ebc4818189b25214bf5d048320418c49d21b1304a149" exitCode=1
Sep 09 20:52:21.599018 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:52:21.598658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr" event={"ID":"a7ed4497-4610-4522-8d49-b360b8fe07ba","Type":"ContainerDied","Data":"da8504646a0c3d4f0419ebc4818189b25214bf5d048320418c49d21b1304a149"}
Sep 09 20:52:21.599018 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:52:21.598972    2578 scope.go:117] "RemoveContainer" containerID="da8504646a0c3d4f0419ebc4818189b25214bf5d048320418c49d21b1304a149"
Sep 09 20:52:22.268856 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:52:22.268821    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:52:22.602664 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:52:22.602636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr" event={"ID":"a7ed4497-4610-4522-8d49-b360b8fe07ba","Type":"ContainerStarted","Data":"6085c4f542e125ad7d29e596d51edd397359c439f174d05e1e45d990cc617a07"}
Sep 09 20:52:22.603009 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:52:22.602834    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:52:22.603499 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:52:22.603479    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf59f8d8-2pprr"
Sep 09 20:52:55.570324 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:52:55.570291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:52:55.570754 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:52:55.570397    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:52:55.570754 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:52:55.570456    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs podName:d29650ed-4c0b-4a60-8128-8b7e95d1a253 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:57.570440668 +0000 UTC m=+374.225220803 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs") pod "network-metrics-daemon-4wr4b" (UID: "d29650ed-4c0b-4a60-8128-8b7e95d1a253") : secret "metrics-daemon-secret" not found
Sep 09 20:53:00.391823 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.391789    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b5nrp"]
Sep 09 20:53:00.393780 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.393766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.400381 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.400354    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:53:00.400381 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.400372    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:53:00.400577 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.400370    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:53:00.400577 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.400479    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ngfdf\""
Sep 09 20:53:00.400717 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.400695    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:53:00.406563 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.406544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqvk\" (UniqueName: \"kubernetes.io/projected/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-api-access-bwqvk\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.406660 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.406571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b022ae00-2834-4f4f-9f87-ba57d67c4702-crio-socket\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.406660 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.406598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.406762 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.406656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.406762 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.406745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b022ae00-2834-4f4f-9f87-ba57d67c4702-data-volume\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.425284 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.425263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b5nrp"]
Sep 09 20:53:00.507348 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.507310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.507508 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.507396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b022ae00-2834-4f4f-9f87-ba57d67c4702-data-volume\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.507508 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.507425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqvk\" (UniqueName: \"kubernetes.io/projected/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-api-access-bwqvk\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.507508 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.507445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b022ae00-2834-4f4f-9f87-ba57d67c4702-crio-socket\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.507508 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.507471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.507508 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:00.507447    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:00.507752 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:00.507544    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:00.507752 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.507546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b022ae00-2834-4f4f-9f87-ba57d67c4702-crio-socket\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.507752 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:00.507577    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls podName:b022ae00-2834-4f4f-9f87-ba57d67c4702 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.007551638 +0000 UTC m=+257.662331790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5nrp" (UID: "b022ae00-2834-4f4f-9f87-ba57d67c4702") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:00.507752 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:00.507602    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm podName:b022ae00-2834-4f4f-9f87-ba57d67c4702 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.007590128 +0000 UTC m=+257.662370270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5nrp" (UID: "b022ae00-2834-4f4f-9f87-ba57d67c4702") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:00.507752 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.507750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b022ae00-2834-4f4f-9f87-ba57d67c4702-data-volume\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:00.517903 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:00.517870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqvk\" (UniqueName: \"kubernetes.io/projected/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-api-access-bwqvk\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:01.011396 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:01.011358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:01.011396 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:01.011397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:01.011650 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:01.011481    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:01.011650 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:01.011504    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:01.011650 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:01.011557    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm podName:b022ae00-2834-4f4f-9f87-ba57d67c4702 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:02.01153948 +0000 UTC m=+258.666319616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5nrp" (UID: "b022ae00-2834-4f4f-9f87-ba57d67c4702") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:01.011650 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:01.011573    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls podName:b022ae00-2834-4f4f-9f87-ba57d67c4702 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:02.011566932 +0000 UTC m=+258.666347068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5nrp" (UID: "b022ae00-2834-4f4f-9f87-ba57d67c4702") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:02.019188 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:02.019117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:02.019188 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:02.019192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:02.019608 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:02.019245    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:02.019608 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:02.019289    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:02.019608 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:02.019319    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm podName:b022ae00-2834-4f4f-9f87-ba57d67c4702 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:04.019302566 +0000 UTC m=+260.674082702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5nrp" (UID: "b022ae00-2834-4f4f-9f87-ba57d67c4702") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:02.019608 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:02.019333    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls podName:b022ae00-2834-4f4f-9f87-ba57d67c4702 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:04.019327511 +0000 UTC m=+260.674107647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5nrp" (UID: "b022ae00-2834-4f4f-9f87-ba57d67c4702") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:03.144134 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:03.144105    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jfdfz_7fd38230-7de5-4519-bf5d-db7c7ea09cc2/dns-node-resolver/0.log"
Sep 09 20:53:03.942118 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:03.942095    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7tx4f_5ca402f4-5c17-48e7-bafd-2fe460254bcf/node-ca/0.log"
Sep 09 20:53:04.034118 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:04.034083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:04.034270 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:04.034127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:04.034270 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:04.034217    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:04.034362 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:04.034275    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm podName:b022ae00-2834-4f4f-9f87-ba57d67c4702 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:08.034259193 +0000 UTC m=+264.689039329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5nrp" (UID: "b022ae00-2834-4f4f-9f87-ba57d67c4702") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:04.034362 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:04.034331    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:04.034448 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:04.034384    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls podName:b022ae00-2834-4f4f-9f87-ba57d67c4702 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:08.034371338 +0000 UTC m=+264.689151474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5nrp" (UID: "b022ae00-2834-4f4f-9f87-ba57d67c4702") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:04.743270 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:04.743244    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mtqlt_6155565e-bace-404d-8d02-daea0e9e697d/serve-healthcheck-canary/0.log"
Sep 09 20:53:08.063729 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:08.063650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:08.064146 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:08.063731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:08.064146 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:08.063786    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:08.064146 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:08.063794    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:08.064146 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:08.063851    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls podName:b022ae00-2834-4f4f-9f87-ba57d67c4702 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.063829189 +0000 UTC m=+272.718609324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5nrp" (UID: "b022ae00-2834-4f4f-9f87-ba57d67c4702") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:08.064146 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:08.063865    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm podName:b022ae00-2834-4f4f-9f87-ba57d67c4702 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.0638596 +0000 UTC m=+272.718639737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5nrp" (UID: "b022ae00-2834-4f4f-9f87-ba57d67c4702") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:16.123416 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:16.123378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:16.123785 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:16.123423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:16.123898 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:16.123876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b022ae00-2834-4f4f-9f87-ba57d67c4702-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:16.125658 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:16.125635    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b022ae00-2834-4f4f-9f87-ba57d67c4702-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5nrp\" (UID: \"b022ae00-2834-4f4f-9f87-ba57d67c4702\") " pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:16.306205 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:16.306156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b5nrp"
Sep 09 20:53:16.577723 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:16.577697    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b5nrp"]
Sep 09 20:53:16.580760 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:53:16.580723    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb022ae00_2834_4f4f_9f87_ba57d67c4702.slice/crio-efd9b1ec84d8b6c5127b73a15b0f12300e2308483f7fe3e1f2b6c5943655b805 WatchSource:0}: Error finding container efd9b1ec84d8b6c5127b73a15b0f12300e2308483f7fe3e1f2b6c5943655b805: Status 404 returned error can't find the container with id efd9b1ec84d8b6c5127b73a15b0f12300e2308483f7fe3e1f2b6c5943655b805
Sep 09 20:53:16.749688 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:16.749656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5nrp" event={"ID":"b022ae00-2834-4f4f-9f87-ba57d67c4702","Type":"ContainerStarted","Data":"dacd5cb39f8e7201adf0d9a65fdbcc3bc8e9f3d400a6d0d1087b3487421f3379"}
Sep 09 20:53:16.749838 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:16.749709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5nrp" event={"ID":"b022ae00-2834-4f4f-9f87-ba57d67c4702","Type":"ContainerStarted","Data":"efd9b1ec84d8b6c5127b73a15b0f12300e2308483f7fe3e1f2b6c5943655b805"}
Sep 09 20:53:17.755233 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:17.755199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5nrp" event={"ID":"b022ae00-2834-4f4f-9f87-ba57d67c4702","Type":"ContainerStarted","Data":"3de444bf73127831e5ae210a6ae2c3caca83371126a7c66c1fd7c3fefd40360e"}
Sep 09 20:53:18.759649 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:18.759562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5nrp" event={"ID":"b022ae00-2834-4f4f-9f87-ba57d67c4702","Type":"ContainerStarted","Data":"d9553200a018ea2a88bdcd3a7475bad3922f4dc4ddc7e62c0690b20b7eb2c4f5"}
Sep 09 20:53:18.787168 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:18.787123    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b5nrp" podStartSLOduration=16.968301178 podStartE2EDuration="18.78710881s" totalImagesPullingTime="1.818807632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:18.787055257 +0000 UTC m=+275.441835414" watchObservedRunningTime="2026-09-09 20:53:18.78710881 +0000 UTC m=+275.441888968"
Sep 09 20:53:24.436984 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:24.436930    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nglhm" podUID="8c27b357-bfd5-4cc0-8da1-8245185755bf"
Sep 09 20:53:24.776913 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:24.776835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nglhm"
Sep 09 20:53:27.704994 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:27.704947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:53:27.707364 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:27.707339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c27b357-bfd5-4cc0-8da1-8245185755bf-metrics-tls\") pod \"dns-default-nglhm\" (UID: \"8c27b357-bfd5-4cc0-8da1-8245185755bf\") " pod="openshift-dns/dns-default-nglhm"
Sep 09 20:53:27.780108 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:27.780081    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5zq52\""
Sep 09 20:53:27.788505 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:27.788486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nglhm"
Sep 09 20:53:28.060374 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:28.060294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nglhm"]
Sep 09 20:53:28.063718 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:53:28.063694    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c27b357_bfd5_4cc0_8da1_8245185755bf.slice/crio-57586d8fa9d96cfb50f9cfcc7bc21d2ac297203d417b80dd69a5d90b091a11ba WatchSource:0}: Error finding container 57586d8fa9d96cfb50f9cfcc7bc21d2ac297203d417b80dd69a5d90b091a11ba: Status 404 returned error can't find the container with id 57586d8fa9d96cfb50f9cfcc7bc21d2ac297203d417b80dd69a5d90b091a11ba
Sep 09 20:53:28.794567 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:28.794525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nglhm" event={"ID":"8c27b357-bfd5-4cc0-8da1-8245185755bf","Type":"ContainerStarted","Data":"57586d8fa9d96cfb50f9cfcc7bc21d2ac297203d417b80dd69a5d90b091a11ba"}
Sep 09 20:53:29.662661 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.662634    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g"]
Sep 09 20:53:29.667201 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.667167    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g"
Sep 09 20:53:29.669133 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.669078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 20:53:29.669133 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.669084    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2d9ff\""
Sep 09 20:53:29.671671 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.671623    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g"]
Sep 09 20:53:29.719857 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.719827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b662ffa4-442d-47e3-86ed-3359fe0b7b03-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wns8g\" (UID: \"b662ffa4-442d-47e3-86ed-3359fe0b7b03\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g"
Sep 09 20:53:29.799720 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.799641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nglhm" event={"ID":"8c27b357-bfd5-4cc0-8da1-8245185755bf","Type":"ContainerStarted","Data":"6f11d2df6cf3b79856cbd3731ccdaebf3f74915e276f563c83048f4e667b15b0"}
Sep 09 20:53:29.799720 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.799696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nglhm" event={"ID":"8c27b357-bfd5-4cc0-8da1-8245185755bf","Type":"ContainerStarted","Data":"3cfbc4845831c36e1a02bdd45a32e8318d622ea63a3b857bc12a051aa684dc57"}
Sep 09 20:53:29.800098 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.799736    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nglhm"
Sep 09 20:53:29.816691 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.816646    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nglhm" podStartSLOduration=251.386072638 podStartE2EDuration="4m12.816629966s" totalImagesPullingTime="1.430557328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:29.816318582 +0000 UTC m=+286.471098741" watchObservedRunningTime="2026-09-09 20:53:29.816629966 +0000 UTC m=+286.471410123"
Sep 09 20:53:29.820553 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.820526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b662ffa4-442d-47e3-86ed-3359fe0b7b03-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wns8g\" (UID: \"b662ffa4-442d-47e3-86ed-3359fe0b7b03\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g"
Sep 09 20:53:29.823027 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.823004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b662ffa4-442d-47e3-86ed-3359fe0b7b03-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wns8g\" (UID: \"b662ffa4-442d-47e3-86ed-3359fe0b7b03\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g"
Sep 09 20:53:29.977589 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:29.977562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g"
Sep 09 20:53:30.244062 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:30.244029    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g"]
Sep 09 20:53:30.249492 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:53:30.249463    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb662ffa4_442d_47e3_86ed_3359fe0b7b03.slice/crio-4cef01f9295d656cd6bfc363029171b603f119e62990cb00198ee8e55c2b00bb WatchSource:0}: Error finding container 4cef01f9295d656cd6bfc363029171b603f119e62990cb00198ee8e55c2b00bb: Status 404 returned error can't find the container with id 4cef01f9295d656cd6bfc363029171b603f119e62990cb00198ee8e55c2b00bb
Sep 09 20:53:30.803881 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:30.803849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g" event={"ID":"b662ffa4-442d-47e3-86ed-3359fe0b7b03","Type":"ContainerStarted","Data":"4cef01f9295d656cd6bfc363029171b603f119e62990cb00198ee8e55c2b00bb"}
Sep 09 20:53:31.808878 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:31.808795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g" event={"ID":"b662ffa4-442d-47e3-86ed-3359fe0b7b03","Type":"ContainerStarted","Data":"91763d05afb2acc9485157fb43e7799b25e456595784f0880167df99bc1ca3b2"}
Sep 09 20:53:31.809248 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:31.809011    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g"
Sep 09 20:53:31.813968 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:31.813939    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g"
Sep 09 20:53:31.828324 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:31.828280    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wns8g" podStartSLOduration=1.7011055050000001 podStartE2EDuration="2.828266043s" totalImagesPullingTime="1.127160538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:31.827439404 +0000 UTC m=+288.482219563" watchObservedRunningTime="2026-09-09 20:53:31.828266043 +0000 UTC m=+288.483046200"
Sep 09 20:53:32.725931 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.725899    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"]
Sep 09 20:53:32.729938 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.729920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:32.732393 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.732373    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:53:32.732622 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.732598    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 20:53:32.732732 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.732626    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 20:53:32.732732 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.732643    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:53:32.733594 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.733576    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:53:32.733886 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.733873    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pzhqd\""
Sep 09 20:53:32.748373 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.748348    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"]
Sep 09 20:53:32.840421 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.840399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:32.840714 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.840464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhch9\" (UniqueName: \"kubernetes.io/projected/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-kube-api-access-nhch9\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:32.840714 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.840502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:32.840714 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.840529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:32.941581 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.941556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhch9\" (UniqueName: \"kubernetes.io/projected/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-kube-api-access-nhch9\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:32.941683 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.941585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:32.941683 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.941604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:32.941683 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.941621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:32.941791 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:32.941718    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 09 20:53:32.941791 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:32.941778    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-prometheus-operator-tls podName:f24a2e20-1e63-4ff3-b2d5-6880648c43bd nodeName:}" failed. No retries permitted until 2026-09-09 20:53:33.441760918 +0000 UTC m=+290.096541060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-8kz5l" (UID: "f24a2e20-1e63-4ff3-b2d5-6880648c43bd") : secret "prometheus-operator-tls" not found
Sep 09 20:53:32.942384 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.942366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:32.943856 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.943839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:32.950018 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:32.949998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhch9\" (UniqueName: \"kubernetes.io/projected/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-kube-api-access-nhch9\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:33.444536 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:33.444489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:33.446928 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:33.446893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f24a2e20-1e63-4ff3-b2d5-6880648c43bd-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-8kz5l\" (UID: \"f24a2e20-1e63-4ff3-b2d5-6880648c43bd\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:33.638471 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:33.638430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"
Sep 09 20:53:33.913485 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:33.913455    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-8kz5l"]
Sep 09 20:53:33.917522 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:53:33.917499    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24a2e20_1e63_4ff3_b2d5_6880648c43bd.slice/crio-e528f3ffa6ed079dc544d043ab8cc6136f84f45606d7963f739a7be9538989cb WatchSource:0}: Error finding container e528f3ffa6ed079dc544d043ab8cc6136f84f45606d7963f739a7be9538989cb: Status 404 returned error can't find the container with id e528f3ffa6ed079dc544d043ab8cc6136f84f45606d7963f739a7be9538989cb
Sep 09 20:53:34.819165 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:34.819124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l" event={"ID":"f24a2e20-1e63-4ff3-b2d5-6880648c43bd","Type":"ContainerStarted","Data":"e528f3ffa6ed079dc544d043ab8cc6136f84f45606d7963f739a7be9538989cb"}
Sep 09 20:53:35.823933 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:35.823894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l" event={"ID":"f24a2e20-1e63-4ff3-b2d5-6880648c43bd","Type":"ContainerStarted","Data":"f76a43ac8487724fa13951b28f711d2f77fe5f441fea585521432832d4aad75c"}
Sep 09 20:53:35.824325 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:35.823956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l" event={"ID":"f24a2e20-1e63-4ff3-b2d5-6880648c43bd","Type":"ContainerStarted","Data":"fa5eba0123e2266a76a9ee82c189af2d4e16840a925521a82ce930645f42ab60"}
Sep 09 20:53:35.848338 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:35.848280    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-8kz5l" podStartSLOduration=2.668351558 podStartE2EDuration="3.848261247s" totalImagesPullingTime="1.179909689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:35.847326425 +0000 UTC m=+292.502106582" watchObservedRunningTime="2026-09-09 20:53:35.848261247 +0000 UTC m=+292.503041406"
Sep 09 20:53:36.838702 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:36.838668    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4wr4b" podUID="d29650ed-4c0b-4a60-8128-8b7e95d1a253"
Sep 09 20:53:38.101656 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.101620    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gjnj2"]
Sep 09 20:53:38.104591 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.104571    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.108330 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.108303    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:53:38.108439 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.108343    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:53:38.108439 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.108418    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:53:38.109161 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.109147    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2sv5v\""
Sep 09 20:53:38.178757 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.178732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-textfile\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.178895 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.178764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-tls\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.178895 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.178782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.178895 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.178802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be4bf9d0-2fc0-4354-8380-81a52e719a5d-root\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.178895 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.178821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.178895 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.178865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be4bf9d0-2fc0-4354-8380-81a52e719a5d-sys\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.179050 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.178982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jppn6\" (UniqueName: \"kubernetes.io/projected/be4bf9d0-2fc0-4354-8380-81a52e719a5d-kube-api-access-jppn6\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.179050 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.179015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be4bf9d0-2fc0-4354-8380-81a52e719a5d-metrics-client-ca\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.179050 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.179032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-wtmp\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280122 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280122 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be4bf9d0-2fc0-4354-8380-81a52e719a5d-sys\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jppn6\" (UniqueName: \"kubernetes.io/projected/be4bf9d0-2fc0-4354-8380-81a52e719a5d-kube-api-access-jppn6\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be4bf9d0-2fc0-4354-8380-81a52e719a5d-metrics-client-ca\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-wtmp\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be4bf9d0-2fc0-4354-8380-81a52e719a5d-sys\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-textfile\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-tls\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280327 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280639 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be4bf9d0-2fc0-4354-8380-81a52e719a5d-root\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280639 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-wtmp\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280639 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:38.280502    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:53:38.280639 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be4bf9d0-2fc0-4354-8380-81a52e719a5d-root\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280639 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:38.280556    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-tls podName:be4bf9d0-2fc0-4354-8380-81a52e719a5d nodeName:}" failed. No retries permitted until 2026-09-09 20:53:38.780538527 +0000 UTC m=+295.435318665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-tls") pod "node-exporter-gjnj2" (UID: "be4bf9d0-2fc0-4354-8380-81a52e719a5d") : secret "node-exporter-tls" not found
Sep 09 20:53:38.280867 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-textfile\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280952 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be4bf9d0-2fc0-4354-8380-81a52e719a5d-metrics-client-ca\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.280987 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.280972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.282484 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.282462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.292431 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.292409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jppn6\" (UniqueName: \"kubernetes.io/projected/be4bf9d0-2fc0-4354-8380-81a52e719a5d-kube-api-access-jppn6\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.783687 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:38.783641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-tls\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:38.783878 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:38.783781    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:53:38.783878 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:53:38.783845    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-tls podName:be4bf9d0-2fc0-4354-8380-81a52e719a5d nodeName:}" failed. No retries permitted until 2026-09-09 20:53:39.783827919 +0000 UTC m=+296.438608054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-tls") pod "node-exporter-gjnj2" (UID: "be4bf9d0-2fc0-4354-8380-81a52e719a5d") : secret "node-exporter-tls" not found
Sep 09 20:53:39.791900 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:39.791867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-tls\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:39.794108 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:39.794082    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be4bf9d0-2fc0-4354-8380-81a52e719a5d-node-exporter-tls\") pod \"node-exporter-gjnj2\" (UID: \"be4bf9d0-2fc0-4354-8380-81a52e719a5d\") " pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:39.806770 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:39.806749    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nglhm"
Sep 09 20:53:39.913878 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:39.913840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gjnj2"
Sep 09 20:53:39.924583 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:53:39.924560    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4bf9d0_2fc0_4354_8380_81a52e719a5d.slice/crio-aa23436df43b1aed3e3ecee471014092b49212a8796bff9b41fa253da89121d4 WatchSource:0}: Error finding container aa23436df43b1aed3e3ecee471014092b49212a8796bff9b41fa253da89121d4: Status 404 returned error can't find the container with id aa23436df43b1aed3e3ecee471014092b49212a8796bff9b41fa253da89121d4
Sep 09 20:53:40.843426 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:40.843394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjnj2" event={"ID":"be4bf9d0-2fc0-4354-8380-81a52e719a5d","Type":"ContainerStarted","Data":"aa23436df43b1aed3e3ecee471014092b49212a8796bff9b41fa253da89121d4"}
Sep 09 20:53:41.848342 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:41.848304    2578 generic.go:356] "Generic (PLEG): container finished" podID="be4bf9d0-2fc0-4354-8380-81a52e719a5d" containerID="11d998e62e5b76cb9219aef06389574e4e84eb66415d56b2c8214be41dd74dc1" exitCode=0
Sep 09 20:53:41.848695 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:41.848362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjnj2" event={"ID":"be4bf9d0-2fc0-4354-8380-81a52e719a5d","Type":"ContainerDied","Data":"11d998e62e5b76cb9219aef06389574e4e84eb66415d56b2c8214be41dd74dc1"}
Sep 09 20:53:42.853340 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:42.853308    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjnj2" event={"ID":"be4bf9d0-2fc0-4354-8380-81a52e719a5d","Type":"ContainerStarted","Data":"fe566780837c0ec70b8699186884c93272c263d315c3d8ed5d65b5e4ffda37d4"}
Sep 09 20:53:42.853682 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:42.853356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjnj2" event={"ID":"be4bf9d0-2fc0-4354-8380-81a52e719a5d","Type":"ContainerStarted","Data":"088692bac4f436b1fcd2766d28c6ad18b2ab2ac6228deab80295880c2ed2c9be"}
Sep 09 20:53:42.881634 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:42.881583    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gjnj2" podStartSLOduration=3.891027461 podStartE2EDuration="4.881566985s" totalImagesPullingTime="930.496068ms" totalInitContainerRuntime="60.043456ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:42.879960904 +0000 UTC m=+299.534741061" watchObservedRunningTime="2026-09-09 20:53:42.881566985 +0000 UTC m=+299.536347144"
Sep 09 20:53:43.509298 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.509262    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86d794f697-zwns6"]
Sep 09 20:53:43.512307 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.512289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.514495 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.514479    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:53:43.514645 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.514630    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f5srx\""
Sep 09 20:53:43.515166 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.515151    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:53:43.515411 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.515396    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:53:43.515484 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.515464    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:53:43.515543 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.515468    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:53:43.515742 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.515726    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:53:43.515839 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.515825    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:53:43.521440 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.521421    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:53:43.523349 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.523330    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d794f697-zwns6"]
Sep 09 20:53:43.626023 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.625992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-oauth-config\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.626023 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.626022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-service-ca\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.626231 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.626046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-console-config\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.626231 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.626119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-trusted-ca-bundle\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.626231 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.626161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-serving-cert\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.626231 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.626204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-oauth-serving-cert\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.626355 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.626247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6w5k\" (UniqueName: \"kubernetes.io/projected/90b1e75a-ae19-4922-9993-037906cc3400-kube-api-access-t6w5k\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.728192 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.728141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-trusted-ca-bundle\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.728370 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.728206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-serving-cert\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.728370 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.728241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-oauth-serving-cert\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.728370 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.728299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6w5k\" (UniqueName: \"kubernetes.io/projected/90b1e75a-ae19-4922-9993-037906cc3400-kube-api-access-t6w5k\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.728370 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.728328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-oauth-config\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.728370 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.728350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-service-ca\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.728628 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.728380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-console-config\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.730669 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.730646    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:53:43.730905 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.730889    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:53:43.731352 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.731305    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:53:43.731601 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.731583    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:53:43.731722 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.731701    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:53:43.735354 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.735338    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:53:43.735906 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.735892    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:53:43.738911 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.738894    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:53:43.739329 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.739296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-console-config\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.739329 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.739296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-oauth-serving-cert\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.739464 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.739337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-trusted-ca-bundle\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.739524 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.739480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-service-ca\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.740960 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.740935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-serving-cert\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.741058 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.741042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-oauth-config\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.749474 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.749455    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:53:43.760250 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.760104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6w5k\" (UniqueName: \"kubernetes.io/projected/90b1e75a-ae19-4922-9993-037906cc3400-kube-api-access-t6w5k\") pod \"console-86d794f697-zwns6\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") " pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:43.824695 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.824676    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f5srx\""
Sep 09 20:53:43.833646 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:43.833588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:44.105196 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:44.105148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d794f697-zwns6"]
Sep 09 20:53:44.108210 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:53:44.108186    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b1e75a_ae19_4922_9993_037906cc3400.slice/crio-5d6324174d25d1d1e0aed6b45f1c07e4971a7edebe6f1b3ead47d09cc9d9741e WatchSource:0}: Error finding container 5d6324174d25d1d1e0aed6b45f1c07e4971a7edebe6f1b3ead47d09cc9d9741e: Status 404 returned error can't find the container with id 5d6324174d25d1d1e0aed6b45f1c07e4971a7edebe6f1b3ead47d09cc9d9741e
Sep 09 20:53:44.861430 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:44.861393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d794f697-zwns6" event={"ID":"90b1e75a-ae19-4922-9993-037906cc3400","Type":"ContainerStarted","Data":"5d6324174d25d1d1e0aed6b45f1c07e4971a7edebe6f1b3ead47d09cc9d9741e"}
Sep 09 20:53:47.873368 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:47.873330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d794f697-zwns6" event={"ID":"90b1e75a-ae19-4922-9993-037906cc3400","Type":"ContainerStarted","Data":"119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a"}
Sep 09 20:53:47.903057 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:47.902944    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86d794f697-zwns6" podStartSLOduration=1.925877267 podStartE2EDuration="4.902926351s" totalImagesPullingTime="2.977049084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:47.902553151 +0000 UTC m=+304.557333332" watchObservedRunningTime="2026-09-09 20:53:47.902926351 +0000 UTC m=+304.557706511"
Sep 09 20:53:48.835483 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:48.835452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:53:53.833943 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:53.833911    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:53.834530 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:53.833981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:53.840014 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:53.839989    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:53:53.894641 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:53:53.894619    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:54:15.552382 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:15.552326    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" podUID="7b9deee8-4ded-4cb6-b2d1-d6a9f686511a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 20:54:25.552964 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:25.552927    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" podUID="7b9deee8-4ded-4cb6-b2d1-d6a9f686511a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 20:54:35.552487 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:35.552443    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" podUID="7b9deee8-4ded-4cb6-b2d1-d6a9f686511a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 20:54:35.552959 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:35.552532    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc"
Sep 09 20:54:35.553134 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:35.553100    2578 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c9ab3d9558efbf7e15c0fa77029b32c7f0df5b6a41d1ede49b95cd197872b2e3"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 20:54:35.553233 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:35.553159    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" podUID="7b9deee8-4ded-4cb6-b2d1-d6a9f686511a" containerName="service-proxy" containerID="cri-o://c9ab3d9558efbf7e15c0fa77029b32c7f0df5b6a41d1ede49b95cd197872b2e3" gracePeriod=30
Sep 09 20:54:36.028609 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:36.028532    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b9deee8-4ded-4cb6-b2d1-d6a9f686511a" containerID="c9ab3d9558efbf7e15c0fa77029b32c7f0df5b6a41d1ede49b95cd197872b2e3" exitCode=2
Sep 09 20:54:36.028777 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:36.028596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" event={"ID":"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a","Type":"ContainerDied","Data":"c9ab3d9558efbf7e15c0fa77029b32c7f0df5b6a41d1ede49b95cd197872b2e3"}
Sep 09 20:54:36.028777 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:36.028653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774b59d9cb-zxcwc" event={"ID":"7b9deee8-4ded-4cb6-b2d1-d6a9f686511a","Type":"ContainerStarted","Data":"eb85a80253f28d5e8e5a66431de2e66d2709e048dfbc989d920b8d0cacbf2847"}
Sep 09 20:54:57.659736 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:57.659698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:54:57.662599 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:57.662564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29650ed-4c0b-4a60-8128-8b7e95d1a253-metrics-certs\") pod \"network-metrics-daemon-4wr4b\" (UID: \"d29650ed-4c0b-4a60-8128-8b7e95d1a253\") " pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:54:57.839002 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:57.838970    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v6qsr\""
Sep 09 20:54:57.845966 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:57.845938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wr4b"
Sep 09 20:54:58.125693 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:58.125662    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4wr4b"]
Sep 09 20:54:58.129512 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:54:58.129484    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29650ed_4c0b_4a60_8128_8b7e95d1a253.slice/crio-6fc1b48a2c31dcb5b5d858f609f59e258d016089cb6b1a8153c89ceef2bada35 WatchSource:0}: Error finding container 6fc1b48a2c31dcb5b5d858f609f59e258d016089cb6b1a8153c89ceef2bada35: Status 404 returned error can't find the container with id 6fc1b48a2c31dcb5b5d858f609f59e258d016089cb6b1a8153c89ceef2bada35
Sep 09 20:54:58.131319 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:58.131292    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:54:59.110845 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:54:59.110810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4wr4b" event={"ID":"d29650ed-4c0b-4a60-8128-8b7e95d1a253","Type":"ContainerStarted","Data":"6fc1b48a2c31dcb5b5d858f609f59e258d016089cb6b1a8153c89ceef2bada35"}
Sep 09 20:55:00.115965 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:00.115929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4wr4b" event={"ID":"d29650ed-4c0b-4a60-8128-8b7e95d1a253","Type":"ContainerStarted","Data":"55bdac6275d94c4f39f196ab30fa8f8efaa8ce5f467c15c5f0412be8ac29a905"}
Sep 09 20:55:00.116354 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:00.115985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4wr4b" event={"ID":"d29650ed-4c0b-4a60-8128-8b7e95d1a253","Type":"ContainerStarted","Data":"9471361c558aabcc2c9fc7c92e255761161bca9a464c768c36cc42a5fa6ed1ff"}
Sep 09 20:55:00.133419 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:00.133370    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4wr4b" podStartSLOduration=375.149578977 podStartE2EDuration="6m16.133353939s" totalImagesPullingTime="983.774962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:00.132755114 +0000 UTC m=+376.787535271" watchObservedRunningTime="2026-09-09 20:55:00.133353939 +0000 UTC m=+376.788134097"
Sep 09 20:55:27.298766 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:27.298730    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d794f697-zwns6"]
Sep 09 20:55:52.321891 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.321833    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86d794f697-zwns6" podUID="90b1e75a-ae19-4922-9993-037906cc3400" containerName="console" containerID="cri-o://119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a" gracePeriod=15
Sep 09 20:55:52.560071 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.560045    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:55:52.651552 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.651520    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-serving-cert\") pod \"90b1e75a-ae19-4922-9993-037906cc3400\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") "
Sep 09 20:55:52.651722 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.651561    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-oauth-config\") pod \"90b1e75a-ae19-4922-9993-037906cc3400\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") "
Sep 09 20:55:52.651722 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.651595    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-trusted-ca-bundle\") pod \"90b1e75a-ae19-4922-9993-037906cc3400\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") "
Sep 09 20:55:52.651722 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.651713    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-service-ca\") pod \"90b1e75a-ae19-4922-9993-037906cc3400\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") "
Sep 09 20:55:52.651889 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.651757    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-console-config\") pod \"90b1e75a-ae19-4922-9993-037906cc3400\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") "
Sep 09 20:55:52.651889 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.651798    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6w5k\" (UniqueName: \"kubernetes.io/projected/90b1e75a-ae19-4922-9993-037906cc3400-kube-api-access-t6w5k\") pod \"90b1e75a-ae19-4922-9993-037906cc3400\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") "
Sep 09 20:55:52.651889 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.651836    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-oauth-serving-cert\") pod \"90b1e75a-ae19-4922-9993-037906cc3400\" (UID: \"90b1e75a-ae19-4922-9993-037906cc3400\") "
Sep 09 20:55:52.652087 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.652060    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "90b1e75a-ae19-4922-9993-037906cc3400" (UID: "90b1e75a-ae19-4922-9993-037906cc3400"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:52.652190 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.652150    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-service-ca" (OuterVolumeSpecName: "service-ca") pod "90b1e75a-ae19-4922-9993-037906cc3400" (UID: "90b1e75a-ae19-4922-9993-037906cc3400"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:52.652238 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.652130    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-console-config" (OuterVolumeSpecName: "console-config") pod "90b1e75a-ae19-4922-9993-037906cc3400" (UID: "90b1e75a-ae19-4922-9993-037906cc3400"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:52.652348 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.652326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "90b1e75a-ae19-4922-9993-037906cc3400" (UID: "90b1e75a-ae19-4922-9993-037906cc3400"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:52.653835 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.653816    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "90b1e75a-ae19-4922-9993-037906cc3400" (UID: "90b1e75a-ae19-4922-9993-037906cc3400"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:52.654133 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.654108    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "90b1e75a-ae19-4922-9993-037906cc3400" (UID: "90b1e75a-ae19-4922-9993-037906cc3400"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:52.654269 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.654130    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b1e75a-ae19-4922-9993-037906cc3400-kube-api-access-t6w5k" (OuterVolumeSpecName: "kube-api-access-t6w5k") pod "90b1e75a-ae19-4922-9993-037906cc3400" (UID: "90b1e75a-ae19-4922-9993-037906cc3400"). InnerVolumeSpecName "kube-api-access-t6w5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:52.752781 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.752753    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 20:55:52.752781 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.752777    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-console-config\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 20:55:52.752781 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.752786    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6w5k\" (UniqueName: \"kubernetes.io/projected/90b1e75a-ae19-4922-9993-037906cc3400-kube-api-access-t6w5k\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 20:55:52.752983 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.752795    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-oauth-serving-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 20:55:52.752983 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.752803    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-serving-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 20:55:52.752983 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.752811    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90b1e75a-ae19-4922-9993-037906cc3400-console-oauth-config\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 20:55:52.752983 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:52.752819    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90b1e75a-ae19-4922-9993-037906cc3400-trusted-ca-bundle\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 20:55:53.291253 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:53.291218    2578 generic.go:356] "Generic (PLEG): container finished" podID="90b1e75a-ae19-4922-9993-037906cc3400" containerID="119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a" exitCode=0
Sep 09 20:55:53.291420 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:53.291262    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d794f697-zwns6"
Sep 09 20:55:53.291420 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:53.291295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d794f697-zwns6" event={"ID":"90b1e75a-ae19-4922-9993-037906cc3400","Type":"ContainerDied","Data":"119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a"}
Sep 09 20:55:53.291420 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:53.291333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d794f697-zwns6" event={"ID":"90b1e75a-ae19-4922-9993-037906cc3400","Type":"ContainerDied","Data":"5d6324174d25d1d1e0aed6b45f1c07e4971a7edebe6f1b3ead47d09cc9d9741e"}
Sep 09 20:55:53.291420 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:53.291353    2578 scope.go:117] "RemoveContainer" containerID="119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a"
Sep 09 20:55:53.299615 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:53.299595    2578 scope.go:117] "RemoveContainer" containerID="119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a"
Sep 09 20:55:53.299919 ip-10-0-136-121 kubenswrapper[2578]: E0909 20:55:53.299901    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a\": container with ID starting with 119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a not found: ID does not exist" containerID="119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a"
Sep 09 20:55:53.299966 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:53.299928    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a"} err="failed to get container status \"119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a\": rpc error: code = NotFound desc = could not find container \"119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a\": container with ID starting with 119d61bed3085a710cf67f8c966cc22dfb0722e39164cc70ab6a4d722a52753a not found: ID does not exist"
Sep 09 20:55:53.312909 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:53.312878    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d794f697-zwns6"]
Sep 09 20:55:53.321561 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:53.321543    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86d794f697-zwns6"]
Sep 09 20:55:53.839640 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:55:53.839608    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b1e75a-ae19-4922-9993-037906cc3400" path="/var/lib/kubelet/pods/90b1e75a-ae19-4922-9993-037906cc3400/volumes"
Sep 09 20:57:07.983663 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:07.983633    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc"]
Sep 09 20:57:07.984062 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:07.983850    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b1e75a-ae19-4922-9993-037906cc3400" containerName="console"
Sep 09 20:57:07.984062 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:07.983860    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b1e75a-ae19-4922-9993-037906cc3400" containerName="console"
Sep 09 20:57:07.984062 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:07.983905    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b1e75a-ae19-4922-9993-037906cc3400" containerName="console"
Sep 09 20:57:07.985503 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:07.985488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc"
Sep 09 20:57:07.987640 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:07.987620    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 20:57:07.987708 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:07.987688    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:57:07.988296 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:07.988284    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gkfvw\""
Sep 09 20:57:07.997691 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:07.997667    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc"]
Sep 09 20:57:08.169949 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:08.169916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db961ba8-1738-4629-8345-eb9409b316d6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pzwsc\" (UID: \"db961ba8-1738-4629-8345-eb9409b316d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc"
Sep 09 20:57:08.169949 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:08.169965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vv2v\" (UniqueName: \"kubernetes.io/projected/db961ba8-1738-4629-8345-eb9409b316d6-kube-api-access-5vv2v\") pod \"openshift-lws-operator-fd8ccff4c-pzwsc\" (UID: \"db961ba8-1738-4629-8345-eb9409b316d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc"
Sep 09 20:57:08.271316 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:08.271237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db961ba8-1738-4629-8345-eb9409b316d6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pzwsc\" (UID: \"db961ba8-1738-4629-8345-eb9409b316d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc"
Sep 09 20:57:08.271316 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:08.271289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vv2v\" (UniqueName: \"kubernetes.io/projected/db961ba8-1738-4629-8345-eb9409b316d6-kube-api-access-5vv2v\") pod \"openshift-lws-operator-fd8ccff4c-pzwsc\" (UID: \"db961ba8-1738-4629-8345-eb9409b316d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc"
Sep 09 20:57:08.271626 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:08.271609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db961ba8-1738-4629-8345-eb9409b316d6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pzwsc\" (UID: \"db961ba8-1738-4629-8345-eb9409b316d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc"
Sep 09 20:57:08.278948 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:08.278922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vv2v\" (UniqueName: \"kubernetes.io/projected/db961ba8-1738-4629-8345-eb9409b316d6-kube-api-access-5vv2v\") pod \"openshift-lws-operator-fd8ccff4c-pzwsc\" (UID: \"db961ba8-1738-4629-8345-eb9409b316d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc"
Sep 09 20:57:08.293438 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:08.293415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc"
Sep 09 20:57:08.589972 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:08.589941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc"]
Sep 09 20:57:08.592835 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:57:08.592809    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb961ba8_1738_4629_8345_eb9409b316d6.slice/crio-91a61572055e5fefd69d39442b1f2314871d4c92ec476ba9d511398b04f9cd34 WatchSource:0}: Error finding container 91a61572055e5fefd69d39442b1f2314871d4c92ec476ba9d511398b04f9cd34: Status 404 returned error can't find the container with id 91a61572055e5fefd69d39442b1f2314871d4c92ec476ba9d511398b04f9cd34
Sep 09 20:57:09.538270 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:09.538228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc" event={"ID":"db961ba8-1738-4629-8345-eb9409b316d6","Type":"ContainerStarted","Data":"91a61572055e5fefd69d39442b1f2314871d4c92ec476ba9d511398b04f9cd34"}
Sep 09 20:57:11.550000 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:11.549906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc" event={"ID":"db961ba8-1738-4629-8345-eb9409b316d6","Type":"ContainerStarted","Data":"099a0cc24f47d886f2b355de9a8d0953675996c119684a6daa7c3ee35f0ad49e"}
Sep 09 20:57:11.571324 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:11.571280    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pzwsc" podStartSLOduration=2.059099548 podStartE2EDuration="4.571266021s" totalImagesPullingTime="2.512166473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:11.570527869 +0000 UTC m=+508.225308027" watchObservedRunningTime="2026-09-09 20:57:11.571266021 +0000 UTC m=+508.226046178"
Sep 09 20:57:16.148314 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.148280    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6jfcn"]
Sep 09 20:57:16.151369 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.151353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6jfcn"
Sep 09 20:57:16.153727 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.153705    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 20:57:16.153727 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.153714    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 20:57:16.154332 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.154318    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6d92j\""
Sep 09 20:57:16.160059 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.160042    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6jfcn"]
Sep 09 20:57:16.334436 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.334409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6sbx\" (UniqueName: \"kubernetes.io/projected/27c0424f-90dc-4b19-8907-9298d50f4a26-kube-api-access-x6sbx\") pod \"cert-manager-55b4d954d5-6jfcn\" (UID: \"27c0424f-90dc-4b19-8907-9298d50f4a26\") " pod="cert-manager/cert-manager-55b4d954d5-6jfcn"
Sep 09 20:57:16.334589 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.334465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27c0424f-90dc-4b19-8907-9298d50f4a26-bound-sa-token\") pod \"cert-manager-55b4d954d5-6jfcn\" (UID: \"27c0424f-90dc-4b19-8907-9298d50f4a26\") " pod="cert-manager/cert-manager-55b4d954d5-6jfcn"
Sep 09 20:57:16.435423 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.435352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6sbx\" (UniqueName: \"kubernetes.io/projected/27c0424f-90dc-4b19-8907-9298d50f4a26-kube-api-access-x6sbx\") pod \"cert-manager-55b4d954d5-6jfcn\" (UID: \"27c0424f-90dc-4b19-8907-9298d50f4a26\") " pod="cert-manager/cert-manager-55b4d954d5-6jfcn"
Sep 09 20:57:16.435423 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.435400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27c0424f-90dc-4b19-8907-9298d50f4a26-bound-sa-token\") pod \"cert-manager-55b4d954d5-6jfcn\" (UID: \"27c0424f-90dc-4b19-8907-9298d50f4a26\") " pod="cert-manager/cert-manager-55b4d954d5-6jfcn"
Sep 09 20:57:16.443556 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.443525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27c0424f-90dc-4b19-8907-9298d50f4a26-bound-sa-token\") pod \"cert-manager-55b4d954d5-6jfcn\" (UID: \"27c0424f-90dc-4b19-8907-9298d50f4a26\") " pod="cert-manager/cert-manager-55b4d954d5-6jfcn"
Sep 09 20:57:16.443656 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.443626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6sbx\" (UniqueName: \"kubernetes.io/projected/27c0424f-90dc-4b19-8907-9298d50f4a26-kube-api-access-x6sbx\") pod \"cert-manager-55b4d954d5-6jfcn\" (UID: \"27c0424f-90dc-4b19-8907-9298d50f4a26\") " pod="cert-manager/cert-manager-55b4d954d5-6jfcn"
Sep 09 20:57:16.460384 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.460368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6jfcn"
Sep 09 20:57:16.726931 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:16.726873    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6jfcn"]
Sep 09 20:57:16.729367 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:57:16.729342    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27c0424f_90dc_4b19_8907_9298d50f4a26.slice/crio-b51f0a3b4f60b57fc71a0b0b8a5d469978104a66fb1ccead05794b9ebe64adde WatchSource:0}: Error finding container b51f0a3b4f60b57fc71a0b0b8a5d469978104a66fb1ccead05794b9ebe64adde: Status 404 returned error can't find the container with id b51f0a3b4f60b57fc71a0b0b8a5d469978104a66fb1ccead05794b9ebe64adde
Sep 09 20:57:17.576745 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:17.576712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6jfcn" event={"ID":"27c0424f-90dc-4b19-8907-9298d50f4a26","Type":"ContainerStarted","Data":"b51f0a3b4f60b57fc71a0b0b8a5d469978104a66fb1ccead05794b9ebe64adde"}
Sep 09 20:57:20.588508 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:20.588475    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6jfcn" event={"ID":"27c0424f-90dc-4b19-8907-9298d50f4a26","Type":"ContainerStarted","Data":"0e063a8865c0d2036fb658fdfb794cd871ac2c0db9151789637e812bb3c543b7"}
Sep 09 20:57:20.604605 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:20.604560    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6jfcn" podStartSLOduration=1.354990391 podStartE2EDuration="4.604544755s" totalImagesPullingTime="3.249554364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:20.603469301 +0000 UTC m=+517.258249472" watchObservedRunningTime="2026-09-09 20:57:20.604544755 +0000 UTC m=+517.259324913"
Sep 09 20:57:24.446508 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.446477    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"]
Sep 09 20:57:24.449548 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.449531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.454808 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.454787    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 20:57:24.455599 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.455297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 20:57:24.456489 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.456474    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xl7pn\""
Sep 09 20:57:24.467279 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.467260    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 20:57:24.478272 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.478248    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"]
Sep 09 20:57:24.488562 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.488541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/769e7d4c-0867-428d-afe4-5e7faedd8efb-cert\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.488638 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.488574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/769e7d4c-0867-428d-afe4-5e7faedd8efb-manager-config\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.488638 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.488627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgbj2\" (UniqueName: \"kubernetes.io/projected/769e7d4c-0867-428d-afe4-5e7faedd8efb-kube-api-access-wgbj2\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.488731 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.488651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/769e7d4c-0867-428d-afe4-5e7faedd8efb-metrics-cert\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.589720 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.589700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbj2\" (UniqueName: \"kubernetes.io/projected/769e7d4c-0867-428d-afe4-5e7faedd8efb-kube-api-access-wgbj2\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.589830 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.589729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/769e7d4c-0867-428d-afe4-5e7faedd8efb-metrics-cert\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.589830 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.589766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/769e7d4c-0867-428d-afe4-5e7faedd8efb-cert\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.589830 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.589793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/769e7d4c-0867-428d-afe4-5e7faedd8efb-manager-config\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.590507 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.590488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/769e7d4c-0867-428d-afe4-5e7faedd8efb-manager-config\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.592162 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.592138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/769e7d4c-0867-428d-afe4-5e7faedd8efb-cert\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.592269 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.592195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/769e7d4c-0867-428d-afe4-5e7faedd8efb-metrics-cert\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.599344 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.599324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgbj2\" (UniqueName: \"kubernetes.io/projected/769e7d4c-0867-428d-afe4-5e7faedd8efb-kube-api-access-wgbj2\") pod \"lws-controller-manager-544fcc9ff7-m5lxx\" (UID: \"769e7d4c-0867-428d-afe4-5e7faedd8efb\") " pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:24.758560 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:24.758464    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:25.025470 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:25.025394    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"]
Sep 09 20:57:25.028351 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:57:25.028323    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769e7d4c_0867_428d_afe4_5e7faedd8efb.slice/crio-864c9e815577b3a0f5f5f78196c6c20ea0b681562dd19cbce23643e773f5bdd7 WatchSource:0}: Error finding container 864c9e815577b3a0f5f5f78196c6c20ea0b681562dd19cbce23643e773f5bdd7: Status 404 returned error can't find the container with id 864c9e815577b3a0f5f5f78196c6c20ea0b681562dd19cbce23643e773f5bdd7
Sep 09 20:57:25.611245 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:25.611184    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx" event={"ID":"769e7d4c-0867-428d-afe4-5e7faedd8efb","Type":"ContainerStarted","Data":"864c9e815577b3a0f5f5f78196c6c20ea0b681562dd19cbce23643e773f5bdd7"}
Sep 09 20:57:26.615908 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:26.615874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx" event={"ID":"769e7d4c-0867-428d-afe4-5e7faedd8efb","Type":"ContainerStarted","Data":"e56c47ad8504d2b0f41d28427908e02bf1fe87bac79a7568bf30bc44eaf4e058"}
Sep 09 20:57:26.616396 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:26.615971    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:26.633019 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:26.632965    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx" podStartSLOduration=1.391245036 podStartE2EDuration="2.632948854s" totalImagesPullingTime="1.241703818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:26.631569726 +0000 UTC m=+523.286349881" watchObservedRunningTime="2026-09-09 20:57:26.632948854 +0000 UTC m=+523.287729013"
Sep 09 20:57:30.110634 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.110604    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"]
Sep 09 20:57:30.113750 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.113734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.116484 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.116460    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 20:57:30.116621 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.116460    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 20:57:30.116621 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.116461    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 20:57:30.116621 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.116497    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 20:57:30.116621 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.116560    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bkhht\""
Sep 09 20:57:30.128758 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.128737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"]
Sep 09 20:57:30.131737 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.131717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eec765b4-c74c-4756-9bdb-f5720d78b735-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.131820 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.131758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97nj\" (UniqueName: \"kubernetes.io/projected/eec765b4-c74c-4756-9bdb-f5720d78b735-kube-api-access-b97nj\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.131820 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.131782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eec765b4-c74c-4756-9bdb-f5720d78b735-manifests\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.131886 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.131844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eec765b4-c74c-4756-9bdb-f5720d78b735-webhook-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.131886 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.131878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eec765b4-c74c-4756-9bdb-f5720d78b735-tmp\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.233105 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.233065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97nj\" (UniqueName: \"kubernetes.io/projected/eec765b4-c74c-4756-9bdb-f5720d78b735-kube-api-access-b97nj\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.233298 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.233131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eec765b4-c74c-4756-9bdb-f5720d78b735-manifests\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.233298 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.233165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eec765b4-c74c-4756-9bdb-f5720d78b735-webhook-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.233298 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.233209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eec765b4-c74c-4756-9bdb-f5720d78b735-tmp\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.233298 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.233244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eec765b4-c74c-4756-9bdb-f5720d78b735-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.233587 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.233556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eec765b4-c74c-4756-9bdb-f5720d78b735-manifests\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.235537 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.235511    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eec765b4-c74c-4756-9bdb-f5720d78b735-tmp\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.235636 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.235549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eec765b4-c74c-4756-9bdb-f5720d78b735-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.235636 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.235572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eec765b4-c74c-4756-9bdb-f5720d78b735-webhook-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.241132 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.241111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97nj\" (UniqueName: \"kubernetes.io/projected/eec765b4-c74c-4756-9bdb-f5720d78b735-kube-api-access-b97nj\") pod \"opendatahub-operator-controller-manager-d75fb655b-w4s89\" (UID: \"eec765b4-c74c-4756-9bdb-f5720d78b735\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.424211 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.424109    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:30.702789 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:30.702708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"]
Sep 09 20:57:30.707002 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:57:30.706967    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec765b4_c74c_4756_9bdb_f5720d78b735.slice/crio-bf83a74ba14502ded6512b8e3c5eab69afa0a5176093cb3fccf6bf3f4b43f83a WatchSource:0}: Error finding container bf83a74ba14502ded6512b8e3c5eab69afa0a5176093cb3fccf6bf3f4b43f83a: Status 404 returned error can't find the container with id bf83a74ba14502ded6512b8e3c5eab69afa0a5176093cb3fccf6bf3f4b43f83a
Sep 09 20:57:31.635278 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:31.635197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89" event={"ID":"eec765b4-c74c-4756-9bdb-f5720d78b735","Type":"ContainerStarted","Data":"bf83a74ba14502ded6512b8e3c5eab69afa0a5176093cb3fccf6bf3f4b43f83a"}
Sep 09 20:57:34.647151 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:34.647116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89" event={"ID":"eec765b4-c74c-4756-9bdb-f5720d78b735","Type":"ContainerStarted","Data":"64b709c7ef95a9e3e59ccf09e1254a88633da8c9ffdaeb594ea4ecc36df3be28"}
Sep 09 20:57:35.652392 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:35.652357    2578 generic.go:356] "Generic (PLEG): container finished" podID="eec765b4-c74c-4756-9bdb-f5720d78b735" containerID="64b709c7ef95a9e3e59ccf09e1254a88633da8c9ffdaeb594ea4ecc36df3be28" exitCode=0
Sep 09 20:57:35.652776 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:35.652409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89" event={"ID":"eec765b4-c74c-4756-9bdb-f5720d78b735","Type":"ContainerDied","Data":"64b709c7ef95a9e3e59ccf09e1254a88633da8c9ffdaeb594ea4ecc36df3be28"}
Sep 09 20:57:36.658121 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:36.658083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89" event={"ID":"eec765b4-c74c-4756-9bdb-f5720d78b735","Type":"ContainerStarted","Data":"ae7d012ee9b485b060c6cda08c58febb1ed9d54cdafbd83849df0030eebb19bf"}
Sep 09 20:57:36.658483 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:36.658236    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:57:36.678501 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:36.678444    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89" podStartSLOduration=2.091390499 podStartE2EDuration="6.678426813s" totalImagesPullingTime="3.505466154s" totalInitContainerRuntime="1.08157016s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:36.676802664 +0000 UTC m=+533.331582844" watchObservedRunningTime="2026-09-09 20:57:36.678426813 +0000 UTC m=+533.333206973"
Sep 09 20:57:37.620994 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:37.620962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-544fcc9ff7-m5lxx"
Sep 09 20:57:47.664785 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:57:47.664751    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-w4s89"
Sep 09 20:58:10.238554 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.238505    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"]
Sep 09 20:58:10.240415 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.240399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.244189 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.244151    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 20:58:10.244385 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.244371    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fgjvh\""
Sep 09 20:58:10.245162 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.245145    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 20:58:10.245352 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.245337    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:58:10.246073 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.246055    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 20:58:10.246359 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.246340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:58:10.246476 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.246456    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:58:10.270464 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.270432    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"]
Sep 09 20:58:10.322854 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.322817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.323051 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.322881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ac86f4-2be9-4205-87b0-f19c31e524f7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.323051 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.322913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42ac86f4-2be9-4205-87b0-f19c31e524f7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.323051 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.322939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.323051 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.322975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.323281 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.323049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqq4q\" (UniqueName: \"kubernetes.io/projected/42ac86f4-2be9-4205-87b0-f19c31e524f7-kube-api-access-cqq4q\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.323281 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.323139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.423822 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.423782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.424013 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.423839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ac86f4-2be9-4205-87b0-f19c31e524f7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.424013 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.423870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42ac86f4-2be9-4205-87b0-f19c31e524f7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.424013 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.423959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.424013 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.424012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.424337 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.424042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqq4q\" (UniqueName: \"kubernetes.io/projected/42ac86f4-2be9-4205-87b0-f19c31e524f7-kube-api-access-cqq4q\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.424337 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.424074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.424919 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.424891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.426306 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.426285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ac86f4-2be9-4205-87b0-f19c31e524f7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.426392 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.426309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.426663 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.426648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42ac86f4-2be9-4205-87b0-f19c31e524f7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.426791 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.426770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.446068 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.446041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ac86f4-2be9-4205-87b0-f19c31e524f7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.457009 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.456972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqq4q\" (UniqueName: \"kubernetes.io/projected/42ac86f4-2be9-4205-87b0-f19c31e524f7-kube-api-access-cqq4q\") pod \"istiod-openshift-gateway-5677ccb49b-vrdgd\" (UID: \"42ac86f4-2be9-4205-87b0-f19c31e524f7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.549243 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.549120    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:10.852165 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:10.852133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"]
Sep 09 20:58:10.855615 ip-10-0-136-121 kubenswrapper[2578]: W0909 20:58:10.855588    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ac86f4_2be9_4205_87b0_f19c31e524f7.slice/crio-eae0104e740b22abd12f0a29e684b7b3d5ac8d575b6bac1d28a2260cc76ad009 WatchSource:0}: Error finding container eae0104e740b22abd12f0a29e684b7b3d5ac8d575b6bac1d28a2260cc76ad009: Status 404 returned error can't find the container with id eae0104e740b22abd12f0a29e684b7b3d5ac8d575b6bac1d28a2260cc76ad009
Sep 09 20:58:11.788477 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:11.788433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd" event={"ID":"42ac86f4-2be9-4205-87b0-f19c31e524f7","Type":"ContainerStarted","Data":"eae0104e740b22abd12f0a29e684b7b3d5ac8d575b6bac1d28a2260cc76ad009"}
Sep 09 20:58:13.124831 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:13.124793    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:58:13.125101 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:13.124861    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:58:13.798435 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:13.798395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd" event={"ID":"42ac86f4-2be9-4205-87b0-f19c31e524f7","Type":"ContainerStarted","Data":"0a2651d6d224f57d3e87b8b188f26fe37636467f68a53f80f79bd1c7cfd81872"}
Sep 09 20:58:13.798642 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:13.798508    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 20:58:14.091343 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:14.091277    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd" podStartSLOduration=1.824230278 podStartE2EDuration="4.091257352s" totalImagesPullingTime="2.267027074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:14.09032521 +0000 UTC m=+570.745105369" watchObservedRunningTime="2026-09-09 20:58:14.091257352 +0000 UTC m=+570.746037511"
Sep 09 20:58:14.804415 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:14.804382    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-vrdgd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:58:14.804843 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:14.804452    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd" podUID="42ac86f4-2be9-4205-87b0-f19c31e524f7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:58:17.804108 ip-10-0-136-121 kubenswrapper[2578]: I0909 20:58:17.804066    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vrdgd"
Sep 09 21:00:14.151108 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.151072    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:00:14.153988 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.153972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg"
Sep 09 21:00:14.156922 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.156902    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xp86x\""
Sep 09 21:00:14.157048 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.156965    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 21:00:14.157548 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.157530    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 21:00:14.157644 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.157567    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 21:00:14.185290 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.185261    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:00:14.237638 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.237606    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:00:14.250510 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.250478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3cdd3e1b-8518-42ed-a96a-19501b6602bf-config-file\") pod \"limitador-limitador-84994f7ffc-6tffg\" (UID: \"3cdd3e1b-8518-42ed-a96a-19501b6602bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg"
Sep 09 21:00:14.250650 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.250535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjm5x\" (UniqueName: \"kubernetes.io/projected/3cdd3e1b-8518-42ed-a96a-19501b6602bf-kube-api-access-gjm5x\") pod \"limitador-limitador-84994f7ffc-6tffg\" (UID: \"3cdd3e1b-8518-42ed-a96a-19501b6602bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg"
Sep 09 21:00:14.351496 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.351457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3cdd3e1b-8518-42ed-a96a-19501b6602bf-config-file\") pod \"limitador-limitador-84994f7ffc-6tffg\" (UID: \"3cdd3e1b-8518-42ed-a96a-19501b6602bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg"
Sep 09 21:00:14.351684 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.351530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjm5x\" (UniqueName: \"kubernetes.io/projected/3cdd3e1b-8518-42ed-a96a-19501b6602bf-kube-api-access-gjm5x\") pod \"limitador-limitador-84994f7ffc-6tffg\" (UID: \"3cdd3e1b-8518-42ed-a96a-19501b6602bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg"
Sep 09 21:00:14.352094 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.352072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3cdd3e1b-8518-42ed-a96a-19501b6602bf-config-file\") pod \"limitador-limitador-84994f7ffc-6tffg\" (UID: \"3cdd3e1b-8518-42ed-a96a-19501b6602bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg"
Sep 09 21:00:14.363266 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.363239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjm5x\" (UniqueName: \"kubernetes.io/projected/3cdd3e1b-8518-42ed-a96a-19501b6602bf-kube-api-access-gjm5x\") pod \"limitador-limitador-84994f7ffc-6tffg\" (UID: \"3cdd3e1b-8518-42ed-a96a-19501b6602bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg"
Sep 09 21:00:14.463303 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.463218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg"
Sep 09 21:00:14.734519 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.734427    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:00:14.737372 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:00:14.737344    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdd3e1b_8518_42ed_a96a_19501b6602bf.slice/crio-6bd33008a29a3a66303372199a657759dbefdd1a6f50b9c32ec646c768a33800 WatchSource:0}: Error finding container 6bd33008a29a3a66303372199a657759dbefdd1a6f50b9c32ec646c768a33800: Status 404 returned error can't find the container with id 6bd33008a29a3a66303372199a657759dbefdd1a6f50b9c32ec646c768a33800
Sep 09 21:00:14.739292 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:14.739274    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:00:15.254180 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:15.254134    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg" event={"ID":"3cdd3e1b-8518-42ed-a96a-19501b6602bf","Type":"ContainerStarted","Data":"6bd33008a29a3a66303372199a657759dbefdd1a6f50b9c32ec646c768a33800"}
Sep 09 21:00:25.298261 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:25.298226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg" event={"ID":"3cdd3e1b-8518-42ed-a96a-19501b6602bf","Type":"ContainerStarted","Data":"4581c1f36e74df48b0bd4eb88ae14c4408e9e29cfde466dc756640101c659d08"}
Sep 09 21:00:25.298728 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:25.298346    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg"
Sep 09 21:00:25.315913 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:25.315846    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg" podStartSLOduration=1.575214672 podStartE2EDuration="11.31583049s" totalImagesPullingTime="9.740615818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:25.31452381 +0000 UTC m=+701.969303969" watchObservedRunningTime="2026-09-09 21:00:25.31583049 +0000 UTC m=+701.970610642"
Sep 09 21:00:36.303366 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:00:36.303288    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tffg"
Sep 09 21:01:33.435067 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:01:33.435032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:02:00.107108 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:00.107074    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:02:15.113720 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:15.113686    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:02:21.101997 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:21.101962    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:02:27.319416 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:27.319382    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:02:42.805098 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.805068    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"]
Sep 09 21:02:42.808096 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.808081    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:42.810236 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.810215    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mnpqm\""
Sep 09 21:02:42.810353 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.810217    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 21:02:42.810758 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.810742    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 21:02:42.810830 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.810769    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 21:02:42.819848 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.819827    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"]
Sep 09 21:02:42.972944 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.972908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:42.973128 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.973009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvq5b\" (UniqueName: \"kubernetes.io/projected/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-kube-api-access-vvq5b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:42.973128 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.973077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:42.973128 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.973116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:42.973313 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.973159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:42.973313 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:42.973242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.073634 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.073557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.073634 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.073595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvq5b\" (UniqueName: \"kubernetes.io/projected/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-kube-api-access-vvq5b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.073634 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.073628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.073891 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.073645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.073891 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.073671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.073891 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.073694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.074045 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.074018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.074102 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.074018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.074102 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.074049    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.075993 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.075969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.076232 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.076213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.081970 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.081948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvq5b\" (UniqueName: \"kubernetes.io/projected/1e54b784-eb60-41ee-aa65-cb7b7b2aaad8-kube-api-access-vvq5b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq\" (UID: \"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.117791 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.117764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:43.389167 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.389135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"]
Sep 09 21:02:43.391883 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:02:43.391854    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e54b784_eb60_41ee_aa65_cb7b7b2aaad8.slice/crio-b66122f6f601e3bd816034335f18902ff10d8d45b6cbee0a53201a254ae0acda WatchSource:0}: Error finding container b66122f6f601e3bd816034335f18902ff10d8d45b6cbee0a53201a254ae0acda: Status 404 returned error can't find the container with id b66122f6f601e3bd816034335f18902ff10d8d45b6cbee0a53201a254ae0acda
Sep 09 21:02:43.845906 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:43.845866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq" event={"ID":"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8","Type":"ContainerStarted","Data":"b66122f6f601e3bd816034335f18902ff10d8d45b6cbee0a53201a254ae0acda"}
Sep 09 21:02:44.303620 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:44.303535    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:02:44.851262 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:44.851220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq" event={"ID":"1e54b784-eb60-41ee-aa65-cb7b7b2aaad8","Type":"ContainerStarted","Data":"7846a86557bedd07ee8a517d35d030e94c5dc7d0077d3ce2a4eadafea09210ef"}
Sep 09 21:02:44.871772 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:44.871722    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq" podStartSLOduration=1.881291051 podStartE2EDuration="2.871706427s" totalImagesPullingTime="990.415376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:44.869889026 +0000 UTC m=+841.524669200" watchObservedRunningTime="2026-09-09 21:02:44.871706427 +0000 UTC m=+841.526486584"
Sep 09 21:02:48.812548 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.812517    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"]
Sep 09 21:02:48.815713 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.815697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.818577 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.818557    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 21:02:48.820681 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.820651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.820789 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.820688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.820854 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.820807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.820915 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.820866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-726gd\" (UniqueName: \"kubernetes.io/projected/509facc5-eeea-41cf-b580-00a0f69763a1-kube-api-access-726gd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.820957 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.820917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509facc5-eeea-41cf-b580-00a0f69763a1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.820957 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.820943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.826930 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.826906    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"]
Sep 09 21:02:48.921355 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.921328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.921487 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.921365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.921487 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.921384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-726gd\" (UniqueName: \"kubernetes.io/projected/509facc5-eeea-41cf-b580-00a0f69763a1-kube-api-access-726gd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.921487 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.921405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509facc5-eeea-41cf-b580-00a0f69763a1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.921487 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.921423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.921487 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.921462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.921774 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.921754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.921884 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.921863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.921919 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.921889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.923541 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.923520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509facc5-eeea-41cf-b580-00a0f69763a1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.928857 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.924167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509facc5-eeea-41cf-b580-00a0f69763a1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:48.950743 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:48.950724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-726gd\" (UniqueName: \"kubernetes.io/projected/509facc5-eeea-41cf-b580-00a0f69763a1-kube-api-access-726gd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7bh5n\" (UID: \"509facc5-eeea-41cf-b580-00a0f69763a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:49.125165 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:49.125129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:49.395685 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:49.395623    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"]
Sep 09 21:02:49.398034 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:02:49.398011    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509facc5_eeea_41cf_b580_00a0f69763a1.slice/crio-e8c3e1d6d4e28fdd22ecb45d1081b37c3f08bff1281be6e4922bdd3635c37039 WatchSource:0}: Error finding container e8c3e1d6d4e28fdd22ecb45d1081b37c3f08bff1281be6e4922bdd3635c37039: Status 404 returned error can't find the container with id e8c3e1d6d4e28fdd22ecb45d1081b37c3f08bff1281be6e4922bdd3635c37039
Sep 09 21:02:49.871789 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:49.871745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n" event={"ID":"509facc5-eeea-41cf-b580-00a0f69763a1","Type":"ContainerStarted","Data":"e1885e34f2b3507af5ae1084ea9561851670f2454a40d5175a85f10d33ae312f"}
Sep 09 21:02:49.872331 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:49.871806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n" event={"ID":"509facc5-eeea-41cf-b580-00a0f69763a1","Type":"ContainerStarted","Data":"e8c3e1d6d4e28fdd22ecb45d1081b37c3f08bff1281be6e4922bdd3635c37039"}
Sep 09 21:02:49.890597 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:49.890550    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n" podStartSLOduration=1.697645665 podStartE2EDuration="1.89053528s" totalImagesPullingTime="192.889615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:49.888909199 +0000 UTC m=+846.543689359" watchObservedRunningTime="2026-09-09 21:02:49.89053528 +0000 UTC m=+846.545315457"
Sep 09 21:02:50.916034 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:50.916003    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:02:53.118168 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:53.118128    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:53.118570 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:53.118206    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:53.130892 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:53.130871    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:53.897780 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:53.897741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq"
Sep 09 21:02:59.125971 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:59.125932    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:59.126438 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:59.125991    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:59.138277 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:59.138257    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:02:59.921318 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:02:59.921290    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7bh5n"
Sep 09 21:03:01.409404 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:03:01.409367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:04:33.205730 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:33.205695    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:04:37.944489 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:37.944457    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"]
Sep 09 21:04:37.947694 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:37.947671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:37.950035 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:37.950008    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ab334d49-openshift-default-dockercfg-nhqjt\""
Sep 09 21:04:37.961222 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:37.961195    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"]
Sep 09 21:04:38.048193 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.048143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istiod-ca-cert\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.048382 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.048221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-podinfo\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.048382 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.048245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-token\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.048382 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.048274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-data\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.048382 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.048363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-credential-socket\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.048527 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.048411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbwr5\" (UniqueName: \"kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-kube-api-access-dbwr5\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.048527 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.048446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-certs\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.048527 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.048491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-envoy\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.048622 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.048542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-socket\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149035 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.148999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-certs\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149252 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-envoy\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149252 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-socket\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149252 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istiod-ca-cert\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149252 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-podinfo\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149252 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-token\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149252 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-data\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149569 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-credential-socket\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149569 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbwr5\" (UniqueName: \"kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-kube-api-access-dbwr5\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149569 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149507    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-certs\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149723 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-socket\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149782 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-credential-socket\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149850 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149827    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-data\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.149906 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.149887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istiod-ca-cert\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.151425 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.151404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-envoy\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.151567 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.151550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-podinfo\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.158657 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.158633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-token\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.158910 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.158894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbwr5\" (UniqueName: \"kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-kube-api-access-dbwr5\") pod \"e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") " pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.258167 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.258090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:38.535263 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:38.535236    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"]
Sep 09 21:04:38.537557 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:04:38.537531    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7678c846_d81a_49d8_bbcb_dbb2d8c051bb.slice/crio-5090c0167ea60ab1c257fe2a7945e987f5b01d0ab38f12382751924cbf52acbe WatchSource:0}: Error finding container 5090c0167ea60ab1c257fe2a7945e987f5b01d0ab38f12382751924cbf52acbe: Status 404 returned error can't find the container with id 5090c0167ea60ab1c257fe2a7945e987f5b01d0ab38f12382751924cbf52acbe
Sep 09 21:04:39.303631 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:39.303597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" event={"ID":"7678c846-d81a-49d8-bbcb-dbb2d8c051bb","Type":"ContainerStarted","Data":"5090c0167ea60ab1c257fe2a7945e987f5b01d0ab38f12382751924cbf52acbe"}
Sep 09 21:04:41.817862 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:41.817824    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:04:41.818142 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:41.817895    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:04:41.818142 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:41.817925    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:04:42.320838 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:42.320805    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" event={"ID":"7678c846-d81a-49d8-bbcb-dbb2d8c051bb","Type":"ContainerStarted","Data":"d561af6e15a1fd94db3013cee485b9d6c74f817f3d66851f52031f131a525559"}
Sep 09 21:04:42.345677 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:42.345630    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" podStartSLOduration=2.067654592 podStartE2EDuration="5.345615749s" totalImagesPullingTime="3.277961157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:42.344369242 +0000 UTC m=+958.999149433" watchObservedRunningTime="2026-09-09 21:04:42.345615749 +0000 UTC m=+959.000395899"
Sep 09 21:04:43.258539 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:43.258505    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:43.264379 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:43.264353    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:43.325254 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:43.325227    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:43.325970 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:43.325954    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:04:44.849062 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.849029    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"]
Sep 09 21:04:44.852679 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.852660    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:44.867979 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.867957    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ab334d49-serving-cert\""
Sep 09 21:04:44.868069 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.868034    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:04:44.868746 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.868731    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-gateway-identity\""
Sep 09 21:04:44.869076 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.869056    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gj4rp\""
Sep 09 21:04:44.869190 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.869057    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 21:04:44.869190 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.869090    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 21:04:44.875708 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.875690    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"]
Sep 09 21:04:44.906255 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.906229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-metrics-certs\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:44.906341 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.906258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-maas-api-tls\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:44.906341 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.906274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nfl\" (UniqueName: \"kubernetes.io/projected/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-kube-api-access-r7nfl\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:44.906418 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:44.906355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-openshift-service-ca\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:45.007001 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:45.006970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-metrics-certs\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:45.007001 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:45.007003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-maas-api-tls\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:45.007268 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:45.007021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nfl\" (UniqueName: \"kubernetes.io/projected/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-kube-api-access-r7nfl\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:45.007268 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:45.007052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-openshift-service-ca\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:45.007670 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:45.007648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-openshift-service-ca\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:45.009582 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:45.009559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-maas-api-tls\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:45.009669 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:45.009566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-metrics-certs\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:45.018924 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:45.018901    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nfl\" (UniqueName: \"kubernetes.io/projected/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-kube-api-access-r7nfl\") pod \"maas-api-e2e-ait-ab334d49-849c665b87-577jj\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:45.162955 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:45.162872    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:45.441074 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:45.441004    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"]
Sep 09 21:04:45.444088 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:04:45.444059    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c85e46a_20dd_411a_99c3_5826e5ecfe8a.slice/crio-b2d6f6a34aceb2e5ecae86c0265e4c0ff08101e3e08f2c0e0af3265e0e76470b WatchSource:0}: Error finding container b2d6f6a34aceb2e5ecae86c0265e4c0ff08101e3e08f2c0e0af3265e0e76470b: Status 404 returned error can't find the container with id b2d6f6a34aceb2e5ecae86c0265e4c0ff08101e3e08f2c0e0af3265e0e76470b
Sep 09 21:04:46.341962 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:46.341922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj" event={"ID":"8c85e46a-20dd-411a-99c3-5826e5ecfe8a","Type":"ContainerStarted","Data":"b2d6f6a34aceb2e5ecae86c0265e4c0ff08101e3e08f2c0e0af3265e0e76470b"}
Sep 09 21:04:47.145301 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.145266    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"]
Sep 09 21:04:47.162634 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.162611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.165217 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.165168    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-80db9b-serving-cert\""
Sep 09 21:04:47.170855 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.170792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"]
Sep 09 21:04:47.225418 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.225386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-maas-api-tls\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.225602 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.225474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-openshift-service-ca\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.225602 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.225507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-metrics-certs\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.225602 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.225545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgs7r\" (UniqueName: \"kubernetes.io/projected/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-kube-api-access-kgs7r\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.326755 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.326720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-openshift-service-ca\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.326903 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.326773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-metrics-certs\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.326903 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.326805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgs7r\" (UniqueName: \"kubernetes.io/projected/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-kube-api-access-kgs7r\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.326903 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.326857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-maas-api-tls\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.327476 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.327443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-openshift-service-ca\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.329581 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.329556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-metrics-certs\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.329683 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.329584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-maas-api-tls\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.336522 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.336498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgs7r\" (UniqueName: \"kubernetes.io/projected/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-kube-api-access-kgs7r\") pod \"maas-api-e2e-autores-80db9b-86679c6bb7-84xch\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.476165 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.476084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:47.975444 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:47.975412    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"]
Sep 09 21:04:47.980552 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:04:47.980525    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7589cd_3cf1_4b2c_b413_8f44ba6060ed.slice/crio-ec3e5edd63ff1439fe1a79551fd53f7a704296ae739d2a06a4a8573f439969b1 WatchSource:0}: Error finding container ec3e5edd63ff1439fe1a79551fd53f7a704296ae739d2a06a4a8573f439969b1: Status 404 returned error can't find the container with id ec3e5edd63ff1439fe1a79551fd53f7a704296ae739d2a06a4a8573f439969b1
Sep 09 21:04:48.352699 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:48.352664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj" event={"ID":"8c85e46a-20dd-411a-99c3-5826e5ecfe8a","Type":"ContainerStarted","Data":"f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed"}
Sep 09 21:04:48.352878 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:48.352742    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:48.353827 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:48.353804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch" event={"ID":"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed","Type":"ContainerStarted","Data":"ec3e5edd63ff1439fe1a79551fd53f7a704296ae739d2a06a4a8573f439969b1"}
Sep 09 21:04:48.379849 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:48.377255    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj" podStartSLOduration=2.080672342 podStartE2EDuration="4.377235308s" totalImagesPullingTime="2.296562966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:48.37512317 +0000 UTC m=+965.029903330" watchObservedRunningTime="2026-09-09 21:04:48.377235308 +0000 UTC m=+965.032015468"
Sep 09 21:04:49.359422 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:49.359385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch" event={"ID":"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed","Type":"ContainerStarted","Data":"cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a"}
Sep 09 21:04:49.360053 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:49.359470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:49.381281 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:49.381225    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch" podStartSLOduration=1.836117757 podStartE2EDuration="2.381206833s" totalImagesPullingTime="545.089076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:49.380914984 +0000 UTC m=+966.035695142" watchObservedRunningTime="2026-09-09 21:04:49.381206833 +0000 UTC m=+966.035986992"
Sep 09 21:04:51.110109 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:51.110074    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:04:54.364636 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:54.364606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:04:55.369514 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:55.369482    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:04:58.345196 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.345097    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"]
Sep 09 21:04:58.348790 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.348769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.351607 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.351589    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-98dad6-openshift-default-dockercfg-pbdfm\""
Sep 09 21:04:58.364163 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.364143    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"]
Sep 09 21:04:58.428658 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.428620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-token\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.429057 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.429035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc426cc6-f400-4bce-a079-a223ae4aad8c-istiod-ca-cert\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.429259 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.429234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-socket\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.429421 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.429406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-envoy\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.429622 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.429603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-data\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.429792 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.429776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-credential-socket\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.429935 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.429920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-certs\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.430050 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.430032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpxtg\" (UniqueName: \"kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-kube-api-access-lpxtg\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.430227 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.430211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-podinfo\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.531144 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.531105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-credential-socket\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.531412 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.531391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-certs\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.531632 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.531614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxtg\" (UniqueName: \"kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-kube-api-access-lpxtg\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.531775 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.531760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-podinfo\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.532397 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.532377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-token\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.532629 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.532611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc426cc6-f400-4bce-a079-a223ae4aad8c-istiod-ca-cert\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.533463 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.533444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-socket\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.533611 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.533595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-envoy\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.533718 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.533705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-data\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.534115 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.534093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-data\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.534247 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.531641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-credential-socket\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.534455 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.534436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-socket\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.534582 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.532100    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-certs\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.534681 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.533391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc426cc6-f400-4bce-a079-a223ae4aad8c-istiod-ca-cert\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.534899 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.534876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-podinfo\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.537434 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.537412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-envoy\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.541910 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.541887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpxtg\" (UniqueName: \"kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-kube-api-access-lpxtg\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.543898 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.543882    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-token\") pod \"e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") " pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.658718 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.658639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:58.943619 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.943532    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"]
Sep 09 21:04:58.947487 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:04:58.947460    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc426cc6_f400_4bce_a079_a223ae4aad8c.slice/crio-f1ad430187efa2bb7306054834c7f84bcbdaa8bdaaec9e166e3d83200dfec215 WatchSource:0}: Error finding container f1ad430187efa2bb7306054834c7f84bcbdaa8bdaaec9e166e3d83200dfec215: Status 404 returned error can't find the container with id f1ad430187efa2bb7306054834c7f84bcbdaa8bdaaec9e166e3d83200dfec215
Sep 09 21:04:58.949735 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.949695    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:04:58.949814 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.949798    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:04:58.949868 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:58.949853    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:04:59.402748 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.402716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" event={"ID":"dc426cc6-f400-4bce-a079-a223ae4aad8c","Type":"ContainerStarted","Data":"3b707f24d8ec46a5ec7e42fc36eb29512165dacabc7a6aa4da127955b9e1b506"}
Sep 09 21:04:59.403107 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.402795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" event={"ID":"dc426cc6-f400-4bce-a079-a223ae4aad8c","Type":"ContainerStarted","Data":"f1ad430187efa2bb7306054834c7f84bcbdaa8bdaaec9e166e3d83200dfec215"}
Sep 09 21:04:59.431554 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.431500    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" podStartSLOduration=1.431468909 podStartE2EDuration="1.431468909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:59.427807552 +0000 UTC m=+976.082587712" watchObservedRunningTime="2026-09-09 21:04:59.431468909 +0000 UTC m=+976.086249067"
Sep 09 21:04:59.659450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.659389    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:04:59.694559 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.694532    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"]
Sep 09 21:04:59.697649 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.697634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.699839 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.699816    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-5927d1b0-kserve-self-signed-certs\""
Sep 09 21:04:59.707433 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.707410    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"]
Sep 09 21:04:59.846648 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.846619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-tmp-dir\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.846648 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.846655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-home\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.846849 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.846713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64af80e-2d08-468b-824a-4aa2f4b57510-tls-certs\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.846849 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.846735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-model-cache\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.846849 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.846818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5zqp\" (UniqueName: \"kubernetes.io/projected/c64af80e-2d08-468b-824a-4aa2f4b57510-kube-api-access-j5zqp\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.846849 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.846845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-dshm\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.947229 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.947140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64af80e-2d08-468b-824a-4aa2f4b57510-tls-certs\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.947229 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.947193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-model-cache\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.947435 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.947247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5zqp\" (UniqueName: \"kubernetes.io/projected/c64af80e-2d08-468b-824a-4aa2f4b57510-kube-api-access-j5zqp\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.947435 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.947267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-dshm\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.947435 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.947288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-tmp-dir\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.947435 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.947316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-home\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.947735 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.947714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-home\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.947815 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.947789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-tmp-dir\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.947871 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.947835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-model-cache\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.949598 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.949564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-dshm\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.949727 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.949708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64af80e-2d08-468b-824a-4aa2f4b57510-tls-certs\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:04:59.956603 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:04:59.956580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5zqp\" (UniqueName: \"kubernetes.io/projected/c64af80e-2d08-468b-824a-4aa2f4b57510-kube-api-access-j5zqp\") pod \"e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") " pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:05:00.007518 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:00.007483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:05:00.283168 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:00.283093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"]
Sep 09 21:05:00.286405 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:05:00.286367    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc64af80e_2d08_468b_824a_4aa2f4b57510.slice/crio-39822e3ece869de27b29bfe48d65d9dafeeb827f0324b3c55ba54184fe4b06d8 WatchSource:0}: Error finding container 39822e3ece869de27b29bfe48d65d9dafeeb827f0324b3c55ba54184fe4b06d8: Status 404 returned error can't find the container with id 39822e3ece869de27b29bfe48d65d9dafeeb827f0324b3c55ba54184fe4b06d8
Sep 09 21:05:00.413032 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:00.412995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7" event={"ID":"c64af80e-2d08-468b-824a-4aa2f4b57510","Type":"ContainerStarted","Data":"637bf9838ab3a49583d3ce4226f68f3c9a85dbfb82f0c08b8d6cf75089b9b13e"}
Sep 09 21:05:00.413409 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:00.413047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7" event={"ID":"c64af80e-2d08-468b-824a-4aa2f4b57510","Type":"ContainerStarted","Data":"39822e3ece869de27b29bfe48d65d9dafeeb827f0324b3c55ba54184fe4b06d8"}
Sep 09 21:05:00.433527 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:00.433478    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7" podStartSLOduration=1.433460015 podStartE2EDuration="1.433460015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:00.432849085 +0000 UTC m=+977.087629244" watchObservedRunningTime="2026-09-09 21:05:00.433460015 +0000 UTC m=+977.088240220"
Sep 09 21:05:00.664647 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:00.664606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:05:01.418122 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:01.418082    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:05:01.418961 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:01.418941    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:05:03.429038 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:03.429005    2578 generic.go:356] "Generic (PLEG): container finished" podID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerID="d561af6e15a1fd94db3013cee485b9d6c74f817f3d66851f52031f131a525559" exitCode=0
Sep 09 21:05:03.429532 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:03.429080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" event={"ID":"7678c846-d81a-49d8-bbcb-dbb2d8c051bb","Type":"ContainerDied","Data":"d561af6e15a1fd94db3013cee485b9d6c74f817f3d66851f52031f131a525559"}
Sep 09 21:05:03.429662 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:03.429643    2578 scope.go:117] "RemoveContainer" containerID="d561af6e15a1fd94db3013cee485b9d6c74f817f3d66851f52031f131a525559"
Sep 09 21:05:03.430644 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:03.430616    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:03.430722 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:03.430686    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:03.430722 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:03.430717    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:04.439761 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:04.439716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" event={"ID":"7678c846-d81a-49d8-bbcb-dbb2d8c051bb","Type":"ContainerStarted","Data":"1bc6d0fb7432a656212ffdb031961baa90565f6bc8403ecaacd379fcffa18bf8"}
Sep 09 21:05:05.258789 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:05.258757    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:05:05.258983 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:05.258959    2578 patch_prober.go:28] interesting pod/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 09 21:05:05.259040 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:05.259020    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 09 21:05:06.259096 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:06.259008    2578 patch_prober.go:28] interesting pod/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 09 21:05:06.259096 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:06.259064    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 09 21:05:07.259197 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:07.259143    2578 patch_prober.go:28] interesting pod/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 09 21:05:07.259581 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:07.259231    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 09 21:05:08.259465 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:08.259424    2578 patch_prober.go:28] interesting pod/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 09 21:05:08.259848 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:08.259485    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 09 21:05:09.258486 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:09.258437    2578 patch_prober.go:28] interesting pod/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 09 21:05:09.258654 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:09.258504    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 09 21:05:10.008415 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:10.008377    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:05:10.008843 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:10.008442    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:05:10.020934 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:10.020909    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:05:10.262572 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:10.262497    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:05:10.262792 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:10.262773    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:05:10.263379 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:10.263362    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:05:10.475904 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:10.475878    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:05:10.713245 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:10.713210    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:05:21.512522 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:21.512434    2578 generic.go:356] "Generic (PLEG): container finished" podID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerID="3b707f24d8ec46a5ec7e42fc36eb29512165dacabc7a6aa4da127955b9e1b506" exitCode=0
Sep 09 21:05:21.512522 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:21.512472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" event={"ID":"dc426cc6-f400-4bce-a079-a223ae4aad8c","Type":"ContainerDied","Data":"3b707f24d8ec46a5ec7e42fc36eb29512165dacabc7a6aa4da127955b9e1b506"}
Sep 09 21:05:21.512998 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:21.512824    2578 scope.go:117] "RemoveContainer" containerID="3b707f24d8ec46a5ec7e42fc36eb29512165dacabc7a6aa4da127955b9e1b506"
Sep 09 21:05:21.513854 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:21.513827    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:21.513906 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:21.513893    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:21.513939 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:21.513923    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:22.521463 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:22.521422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" event={"ID":"dc426cc6-f400-4bce-a079-a223ae4aad8c","Type":"ContainerStarted","Data":"f32808716fa01a069019ec7eb69d4d456eb56e6c8d7111333cf50c38fcb96b85"}
Sep 09 21:05:22.659220 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:22.659157    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:05:22.659438 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:22.659411    2578 patch_prober.go:28] interesting pod/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 09 21:05:22.659486 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:22.659466    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 09 21:05:23.659043 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:23.659007    2578 patch_prober.go:28] interesting pod/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 09 21:05:23.659459 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:23.659062    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 09 21:05:24.659006 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:24.658970    2578 patch_prober.go:28] interesting pod/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 09 21:05:24.659166 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:24.659022    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 09 21:05:25.659565 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:25.659532    2578 patch_prober.go:28] interesting pod/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 09 21:05:25.660019 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:25.659592    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 09 21:05:25.809369 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:25.809336    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"]
Sep 09 21:05:25.809545 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:25.809523    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj" podUID="8c85e46a-20dd-411a-99c3-5826e5ecfe8a" containerName="maas-api" containerID="cri-o://f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed" gracePeriod=30
Sep 09 21:05:25.971690 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:25.971654    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:05:26.088812 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.088789    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:05:26.182303 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.182204    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7nfl\" (UniqueName: \"kubernetes.io/projected/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-kube-api-access-r7nfl\") pod \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") "
Sep 09 21:05:26.182303 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.182262    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-openshift-service-ca\") pod \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") "
Sep 09 21:05:26.182528 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.182358    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-maas-api-tls\") pod \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") "
Sep 09 21:05:26.182528 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.182437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-metrics-certs\") pod \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\" (UID: \"8c85e46a-20dd-411a-99c3-5826e5ecfe8a\") "
Sep 09 21:05:26.182634 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.182564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c85e46a-20dd-411a-99c3-5826e5ecfe8a" (UID: "8c85e46a-20dd-411a-99c3-5826e5ecfe8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:05:26.182755 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.182733    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:26.184609 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.184585    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8c85e46a-20dd-411a-99c3-5826e5ecfe8a" (UID: "8c85e46a-20dd-411a-99c3-5826e5ecfe8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:05:26.184692 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.184605    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-kube-api-access-r7nfl" (OuterVolumeSpecName: "kube-api-access-r7nfl") pod "8c85e46a-20dd-411a-99c3-5826e5ecfe8a" (UID: "8c85e46a-20dd-411a-99c3-5826e5ecfe8a"). InnerVolumeSpecName "kube-api-access-r7nfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:05:26.184692 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.184605    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c85e46a-20dd-411a-99c3-5826e5ecfe8a" (UID: "8c85e46a-20dd-411a-99c3-5826e5ecfe8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:05:26.283959 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.283925    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7nfl\" (UniqueName: \"kubernetes.io/projected/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-kube-api-access-r7nfl\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:26.283959 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.283956    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:26.283959 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.283966    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c85e46a-20dd-411a-99c3-5826e5ecfe8a-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:26.344197 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.335804    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"]
Sep 09 21:05:26.344197 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.336384    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy" containerID="cri-o://1bc6d0fb7432a656212ffdb031961baa90565f6bc8403ecaacd379fcffa18bf8" gracePeriod=30
Sep 09 21:05:26.549704 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.549617    2578 generic.go:356] "Generic (PLEG): container finished" podID="8c85e46a-20dd-411a-99c3-5826e5ecfe8a" containerID="f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed" exitCode=0
Sep 09 21:05:26.549704 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.549657    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"
Sep 09 21:05:26.549704 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.549668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj" event={"ID":"8c85e46a-20dd-411a-99c3-5826e5ecfe8a","Type":"ContainerDied","Data":"f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed"}
Sep 09 21:05:26.549927 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.549723    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj" event={"ID":"8c85e46a-20dd-411a-99c3-5826e5ecfe8a","Type":"ContainerDied","Data":"b2d6f6a34aceb2e5ecae86c0265e4c0ff08101e3e08f2c0e0af3265e0e76470b"}
Sep 09 21:05:26.549927 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.549751    2578 scope.go:117] "RemoveContainer" containerID="f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed"
Sep 09 21:05:26.559691 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.559665    2578 scope.go:117] "RemoveContainer" containerID="f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed"
Sep 09 21:05:26.560096 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:05:26.560051    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed\": container with ID starting with f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed not found: ID does not exist" containerID="f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed"
Sep 09 21:05:26.560264 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.560110    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed"} err="failed to get container status \"f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed\": rpc error: code = NotFound desc = could not find container \"f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed\": container with ID starting with f6208f25333e90c75c9fae2968dedd21c5a412ac52da239804fc1b010ef8dfed not found: ID does not exist"
Sep 09 21:05:26.576220 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.576196    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"]
Sep 09 21:05:26.580043 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.580021    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab334d49-849c665b87-577jj"]
Sep 09 21:05:26.659936 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.659892    2578 patch_prober.go:28] interesting pod/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 09 21:05:26.660359 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:26.659955    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 09 21:05:27.305460 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.305430    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"]
Sep 09 21:05:27.305744 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.305732    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c85e46a-20dd-411a-99c3-5826e5ecfe8a" containerName="maas-api"
Sep 09 21:05:27.305802 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.305745    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c85e46a-20dd-411a-99c3-5826e5ecfe8a" containerName="maas-api"
Sep 09 21:05:27.305840 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.305803    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c85e46a-20dd-411a-99c3-5826e5ecfe8a" containerName="maas-api"
Sep 09 21:05:27.308590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.308572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.311874 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.311853    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e62fbb04-gw-openshift-default-dockercfg-v4br7\""
Sep 09 21:05:27.326347 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.326320    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"]
Sep 09 21:05:27.394120 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.394089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istiod-ca-cert\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.394322 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.394128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-data\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.394322 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.394155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-token\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.394322 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.394275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-socket\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.394322 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.394296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hht\" (UniqueName: \"kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-kube-api-access-g8hht\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.394463 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.394322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-credential-socket\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.394463 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.394345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-podinfo\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.394463 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.394361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-certs\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.394463 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.394379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-envoy\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.495550 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.495507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istiod-ca-cert\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.495722 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.495564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-data\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.495722 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.495610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-token\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.495722 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.495647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-socket\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.495905 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.495820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hht\" (UniqueName: \"kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-kube-api-access-g8hht\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.495905 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.495872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-credential-socket\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.496022 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.495905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-podinfo\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.496022 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.495935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-certs\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.496022 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.495959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-envoy\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.496319 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.496295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-credential-socket\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.496400 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.496296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-certs\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.496400 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.496364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istiod-ca-cert\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.496508 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.496487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-data\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.496547 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.496513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-socket\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.498167 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.498147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-envoy\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.498725 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.498694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-podinfo\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.504715 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.504693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-token\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.504831 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.504808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hht\" (UniqueName: \"kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-kube-api-access-g8hht\") pod \"e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") " pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.627430 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.627390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:27.668057 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.668025    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:05:27.668601 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.668569    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:05:27.669525 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.669498    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:05:27.841723 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.841691    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c85e46a-20dd-411a-99c3-5826e5ecfe8a" path="/var/lib/kubelet/pods/8c85e46a-20dd-411a-99c3-5826e5ecfe8a/volumes"
Sep 09 21:05:27.902935 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.902913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"]
Sep 09 21:05:27.905323 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:05:27.905290    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3de8a3_4e99_4a25_a2e0_58ca9ed1919e.slice/crio-704574503eeb5792b7166a4f52450daaee4eab11e8603eeba0ec56213bff535e WatchSource:0}: Error finding container 704574503eeb5792b7166a4f52450daaee4eab11e8603eeba0ec56213bff535e: Status 404 returned error can't find the container with id 704574503eeb5792b7166a4f52450daaee4eab11e8603eeba0ec56213bff535e
Sep 09 21:05:27.907420 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.907386    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:27.907515 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.907448    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:27.907515 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:27.907475    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:28.562751 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:28.562719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" event={"ID":"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e","Type":"ContainerStarted","Data":"f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90"}
Sep 09 21:05:28.562893 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:28.562770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" event={"ID":"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e","Type":"ContainerStarted","Data":"704574503eeb5792b7166a4f52450daaee4eab11e8603eeba0ec56213bff535e"}
Sep 09 21:05:28.586039 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:28.585976    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" podStartSLOduration=1.585957544 podStartE2EDuration="1.585957544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:28.584749568 +0000 UTC m=+1005.239529725" watchObservedRunningTime="2026-09-09 21:05:28.585957544 +0000 UTC m=+1005.240737703"
Sep 09 21:05:28.628313 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:28.628285    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:28.633064 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:28.633042    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:29.567209 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:29.567158    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:29.568031 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:29.568011    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:31.578534 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.578504    2578 generic.go:356] "Generic (PLEG): container finished" podID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerID="1bc6d0fb7432a656212ffdb031961baa90565f6bc8403ecaacd379fcffa18bf8" exitCode=0
Sep 09 21:05:31.578833 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.578579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" event={"ID":"7678c846-d81a-49d8-bbcb-dbb2d8c051bb","Type":"ContainerDied","Data":"1bc6d0fb7432a656212ffdb031961baa90565f6bc8403ecaacd379fcffa18bf8"}
Sep 09 21:05:31.578833 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.578630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt" event={"ID":"7678c846-d81a-49d8-bbcb-dbb2d8c051bb","Type":"ContainerDied","Data":"5090c0167ea60ab1c257fe2a7945e987f5b01d0ab38f12382751924cbf52acbe"}
Sep 09 21:05:31.578833 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.578646    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5090c0167ea60ab1c257fe2a7945e987f5b01d0ab38f12382751924cbf52acbe"
Sep 09 21:05:31.578833 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.578659    2578 scope.go:117] "RemoveContainer" containerID="d561af6e15a1fd94db3013cee485b9d6c74f817f3d66851f52031f131a525559"
Sep 09 21:05:31.579025 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.578938    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:05:31.633447 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633382    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-token\") pod \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") "
Sep 09 21:05:31.633447 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633422    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-data\") pod \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") "
Sep 09 21:05:31.633621 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istiod-ca-cert\") pod \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") "
Sep 09 21:05:31.633621 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbwr5\" (UniqueName: \"kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-kube-api-access-dbwr5\") pod \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") "
Sep 09 21:05:31.633621 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633523    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-socket\") pod \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") "
Sep 09 21:05:31.633621 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633552    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-credential-socket\") pod \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") "
Sep 09 21:05:31.633621 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633601    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-podinfo\") pod \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") "
Sep 09 21:05:31.633855 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633647    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-certs\") pod \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") "
Sep 09 21:05:31.633855 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633678    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-envoy\") pod \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\" (UID: \"7678c846-d81a-49d8-bbcb-dbb2d8c051bb\") "
Sep 09 21:05:31.633855 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633753    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "7678c846-d81a-49d8-bbcb-dbb2d8c051bb" (UID: "7678c846-d81a-49d8-bbcb-dbb2d8c051bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:31.633855 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7678c846-d81a-49d8-bbcb-dbb2d8c051bb" (UID: "7678c846-d81a-49d8-bbcb-dbb2d8c051bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:31.634053 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.633882    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7678c846-d81a-49d8-bbcb-dbb2d8c051bb" (UID: "7678c846-d81a-49d8-bbcb-dbb2d8c051bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:05:31.634053 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.634037    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7678c846-d81a-49d8-bbcb-dbb2d8c051bb" (UID: "7678c846-d81a-49d8-bbcb-dbb2d8c051bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:31.634156 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.634125    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7678c846-d81a-49d8-bbcb-dbb2d8c051bb" (UID: "7678c846-d81a-49d8-bbcb-dbb2d8c051bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:31.634252 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.634235    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.634310 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.634261    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.634310 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.634275    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.634310 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.634299    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.634498 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.634314    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.635818 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.635795    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-kube-api-access-dbwr5" (OuterVolumeSpecName: "kube-api-access-dbwr5") pod "7678c846-d81a-49d8-bbcb-dbb2d8c051bb" (UID: "7678c846-d81a-49d8-bbcb-dbb2d8c051bb"). InnerVolumeSpecName "kube-api-access-dbwr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:05:31.635905 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.635821    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7678c846-d81a-49d8-bbcb-dbb2d8c051bb" (UID: "7678c846-d81a-49d8-bbcb-dbb2d8c051bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:05:31.635965 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.635936    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "7678c846-d81a-49d8-bbcb-dbb2d8c051bb" (UID: "7678c846-d81a-49d8-bbcb-dbb2d8c051bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:05:31.636137 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.636121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7678c846-d81a-49d8-bbcb-dbb2d8c051bb" (UID: "7678c846-d81a-49d8-bbcb-dbb2d8c051bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:31.734780 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.734751    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.734780 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.734776    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbwr5\" (UniqueName: \"kubernetes.io/projected/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-kube-api-access-dbwr5\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.734780 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.734786    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.734965 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:31.734795    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7678c846-d81a-49d8-bbcb-dbb2d8c051bb-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:05:32.583804 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:32.583772    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"
Sep 09 21:05:32.646234 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:32.646193    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"]
Sep 09 21:05:32.648311 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:32.648288    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ab334d49-openshift-default-67c8489f76-cnhjt"]
Sep 09 21:05:33.840735 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:33.840703    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" path="/var/lib/kubelet/pods/7678c846-d81a-49d8-bbcb-dbb2d8c051bb/volumes"
Sep 09 21:05:42.523397 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:42.523358    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:05:52.157858 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:05:52.157834    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@2-12109-0.service\": RecentStats: unable to find data in memory cache]"
Sep 09 21:05:52.158166 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:05:52.157833    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3de8a3_4e99_4a25_a2e0_58ca9ed1919e.slice/crio-conmon-f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3de8a3_4e99_4a25_a2e0_58ca9ed1919e.slice/crio-f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@2-12109-0.service\": RecentStats: unable to find data in memory cache]"
Sep 09 21:05:52.675816 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:52.675785    2578 generic.go:356] "Generic (PLEG): container finished" podID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerID="f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90" exitCode=0
Sep 09 21:05:52.675991 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:52.675822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" event={"ID":"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e","Type":"ContainerDied","Data":"f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90"}
Sep 09 21:05:52.676163 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:52.676150    2578 scope.go:117] "RemoveContainer" containerID="f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90"
Sep 09 21:05:52.677251 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:52.677211    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:52.677353 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:52.677294    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:52.677353 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:52.677347    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:05:53.682324 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:53.682288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" event={"ID":"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e","Type":"ContainerStarted","Data":"6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d"}
Sep 09 21:05:54.628106 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:54.628071    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:54.628500 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:54.628402    2578 patch_prober.go:28] interesting pod/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 09 21:05:54.628656 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:54.628634    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 09 21:05:55.048622 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.048539    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"]
Sep 09 21:05:55.048975 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.048835    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy"
Sep 09 21:05:55.048975 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.048846    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy"
Sep 09 21:05:55.048975 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.048913    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy"
Sep 09 21:05:55.050951 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.050935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"
Sep 09 21:05:55.053149 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.053125    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6l2wq\""
Sep 09 21:05:55.058998 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.058974    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"]
Sep 09 21:05:55.151625 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.151587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a950021-afba-4545-90f4-99383f34d5dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f\" (UID: \"5a950021-afba-4545-90f4-99383f34d5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"
Sep 09 21:05:55.253121 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.253078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a950021-afba-4545-90f4-99383f34d5dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f\" (UID: \"5a950021-afba-4545-90f4-99383f34d5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"
Sep 09 21:05:55.253776 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.253751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a950021-afba-4545-90f4-99383f34d5dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f\" (UID: \"5a950021-afba-4545-90f4-99383f34d5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"
Sep 09 21:05:55.361415 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.361383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"
Sep 09 21:05:55.628157 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.628064    2578 patch_prober.go:28] interesting pod/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 09 21:05:55.628346 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.628148    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 09 21:05:55.667699 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:05:55.667667    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a950021_afba_4545_90f4_99383f34d5dc.slice/crio-14d30310b91c08a342421a097d6746e3dae97beb7c59a6f7906d136f24adf5cc WatchSource:0}: Error finding container 14d30310b91c08a342421a097d6746e3dae97beb7c59a6f7906d136f24adf5cc: Status 404 returned error can't find the container with id 14d30310b91c08a342421a097d6746e3dae97beb7c59a6f7906d136f24adf5cc
Sep 09 21:05:55.669916 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.669897    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:05:55.670495 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.670396    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"]
Sep 09 21:05:55.692350 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:55.692296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f" event={"ID":"5a950021-afba-4545-90f4-99383f34d5dc","Type":"ContainerStarted","Data":"14d30310b91c08a342421a097d6746e3dae97beb7c59a6f7906d136f24adf5cc"}
Sep 09 21:05:56.627839 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:56.627785    2578 patch_prober.go:28] interesting pod/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 09 21:05:56.628834 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:56.628391    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 09 21:05:57.013931 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.013708    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"]
Sep 09 21:05:57.017201 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.014239    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy"
Sep 09 21:05:57.017201 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.014262    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy"
Sep 09 21:05:57.017201 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.014351    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7678c846-d81a-49d8-bbcb-dbb2d8c051bb" containerName="istio-proxy"
Sep 09 21:05:57.029469 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.029341    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"]
Sep 09 21:05:57.030045 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.029453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.035103 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.034304    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-5927d1b0-kserve-self-signed-certs\""
Sep 09 21:05:57.169795 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.169750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6f15f4-1a14-4423-a1dc-060a01698302-tls-certs\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.169971 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.169901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-home\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.169971 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.169934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcpmr\" (UniqueName: \"kubernetes.io/projected/ac6f15f4-1a14-4423-a1dc-060a01698302-kube-api-access-lcpmr\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.169971 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.169969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-dshm\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.170165 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.170041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-model-cache\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.170234 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.170160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-tmp-dir\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.271253 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.271150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-home\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.271253 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.271215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpmr\" (UniqueName: \"kubernetes.io/projected/ac6f15f4-1a14-4423-a1dc-060a01698302-kube-api-access-lcpmr\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.271253 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.271249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-dshm\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.271532 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.271287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-model-cache\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.271532 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.271343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-tmp-dir\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.271532 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.271374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6f15f4-1a14-4423-a1dc-060a01698302-tls-certs\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.271849 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.271797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-home\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.271849 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.271833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-model-cache\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.272028 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.271900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-tmp-dir\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.276492 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.276464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-dshm\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.276759 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.276732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6f15f4-1a14-4423-a1dc-060a01698302-tls-certs\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.279603 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.279584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpmr\" (UniqueName: \"kubernetes.io/projected/ac6f15f4-1a14-4423-a1dc-060a01698302-kube-api-access-lcpmr\") pod \"e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") " pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.346193 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.346148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:05:57.636194 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.631211    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"]
Sep 09 21:05:57.636194 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.631979    2578 patch_prober.go:28] interesting pod/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 09 21:05:57.636194 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.632029    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 09 21:05:57.702973 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:57.702922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v" event={"ID":"ac6f15f4-1a14-4423-a1dc-060a01698302","Type":"ContainerStarted","Data":"53fff6ec198a10f48b2b9c44d880feeaf3818488cec765f0e80ef284ca5025d9"}
Sep 09 21:05:58.628533 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:58.628493    2578 patch_prober.go:28] interesting pod/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 09 21:05:58.628711 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:58.628556    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 09 21:05:58.710411 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:58.710367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v" event={"ID":"ac6f15f4-1a14-4423-a1dc-060a01698302","Type":"ContainerStarted","Data":"3072bc1ebb790bc39e0e95a894105f46635393cc6d26ff64e0ca5b9ed3e05796"}
Sep 09 21:05:58.734646 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:58.734585    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v" podStartSLOduration=2.734561111 podStartE2EDuration="2.734561111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:58.731782163 +0000 UTC m=+1035.386562361" watchObservedRunningTime="2026-09-09 21:05:58.734561111 +0000 UTC m=+1035.389341270"
Sep 09 21:05:59.567664 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:59.567634    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:59.632693 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:59.632671    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:05:59.716404 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:59.716371    2578 generic.go:356] "Generic (PLEG): container finished" podID="5a950021-afba-4545-90f4-99383f34d5dc" containerID="aab8ab60301aef256bf47d16ce1a92bec82d986e76650b112b1cd6d5d9dba887" exitCode=0
Sep 09 21:05:59.716745 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:59.716458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f" event={"ID":"5a950021-afba-4545-90f4-99383f34d5dc","Type":"ContainerDied","Data":"aab8ab60301aef256bf47d16ce1a92bec82d986e76650b112b1cd6d5d9dba887"}
Sep 09 21:05:59.717549 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:05:59.717530    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:06:00.850002 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:00.849981    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"
Sep 09 21:06:01.008112 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:01.008028    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a950021-afba-4545-90f4-99383f34d5dc-maas-api-service-ca\") pod \"5a950021-afba-4545-90f4-99383f34d5dc\" (UID: \"5a950021-afba-4545-90f4-99383f34d5dc\") "
Sep 09 21:06:01.008446 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:01.008414    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a950021-afba-4545-90f4-99383f34d5dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a950021-afba-4545-90f4-99383f34d5dc" (UID: "5a950021-afba-4545-90f4-99383f34d5dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:01.109482 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:01.109454    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a950021-afba-4545-90f4-99383f34d5dc-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:01.726938 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:01.726906    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"
Sep 09 21:06:01.726938 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:01.726930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f" event={"ID":"5a950021-afba-4545-90f4-99383f34d5dc","Type":"ContainerDied","Data":"14d30310b91c08a342421a097d6746e3dae97beb7c59a6f7906d136f24adf5cc"}
Sep 09 21:06:01.727138 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:01.726973    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14d30310b91c08a342421a097d6746e3dae97beb7c59a6f7906d136f24adf5cc"
Sep 09 21:06:02.610347 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:02.610304    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:06:07.347246 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:07.347162    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:06:07.347737 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:07.347267    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:06:07.359676 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:07.359652    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:06:07.764224 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:07.764133    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:06:11.796261 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:11.796225    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"]
Sep 09 21:06:11.800724 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:11.800703    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e62fbb04-2pt8f"]
Sep 09 21:06:11.840968 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:11.840930    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a950021-afba-4545-90f4-99383f34d5dc" path="/var/lib/kubelet/pods/5a950021-afba-4545-90f4-99383f34d5dc/volumes"
Sep 09 21:06:11.937746 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:11.937686    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"]
Sep 09 21:06:11.938074 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:11.938038    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy" containerID="cri-o://6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d" gracePeriod=30
Sep 09 21:06:12.668998 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.668964    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"]
Sep 09 21:06:12.669392 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.669373    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a950021-afba-4545-90f4-99383f34d5dc" containerName="revoke-keys"
Sep 09 21:06:12.669392 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.669392    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a950021-afba-4545-90f4-99383f34d5dc" containerName="revoke-keys"
Sep 09 21:06:12.669523 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.669457    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a950021-afba-4545-90f4-99383f34d5dc" containerName="revoke-keys"
Sep 09 21:06:12.675359 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.675339    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.677496 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.677477    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-857dd97d-openshift-default-dockercfg-4hnlh\""
Sep 09 21:06:12.685882 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.685861    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"]
Sep 09 21:06:12.701159 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.701133    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istiod-ca-cert\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.701296 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.701215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-token\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.701296 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.701276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-credential-socket\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.701415 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.701318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-envoy\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.701415 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.701355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-certs\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.701415 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.701408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwjr\" (UniqueName: \"kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-kube-api-access-8zwjr\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.701554 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.701446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-podinfo\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.701554 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.701495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-socket\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.701634 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.701558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-data\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.802337 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-token\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.802337 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-credential-socket\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.802828 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-envoy\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.802828 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-certs\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.802828 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwjr\" (UniqueName: \"kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-kube-api-access-8zwjr\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.802828 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-podinfo\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.802828 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-socket\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.802828 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-data\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.802828 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istiod-ca-cert\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.802828 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-credential-socket\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.803145 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-certs\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.803145 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.802999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-data\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.803145 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.803079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-socket\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.803306 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.803289    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istiod-ca-cert\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.804704 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.804677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-envoy\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.805036 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.805014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-podinfo\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.814047 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.814023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-token\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.814258 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.814241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwjr\" (UniqueName: \"kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-kube-api-access-8zwjr\") pod \"e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") " pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:12.985916 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:12.985841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:13.261405 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:13.261373    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"]
Sep 09 21:06:13.263786 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:06:13.263761    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd569026a_daa3_47c4_b080_6c4ad7d3f31a.slice/crio-5e4a0b592a64c0b14eb30ec81ba2f6592f085cf945ca846eac0eba0c9c50d192 WatchSource:0}: Error finding container 5e4a0b592a64c0b14eb30ec81ba2f6592f085cf945ca846eac0eba0c9c50d192: Status 404 returned error can't find the container with id 5e4a0b592a64c0b14eb30ec81ba2f6592f085cf945ca846eac0eba0c9c50d192
Sep 09 21:06:13.266060 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:13.266026    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:06:13.266136 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:13.266090    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:06:13.266136 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:13.266123    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:06:13.780826 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:13.780790    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" event={"ID":"d569026a-daa3-47c4-b080-6c4ad7d3f31a","Type":"ContainerStarted","Data":"0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a"}
Sep 09 21:06:13.781292 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:13.780840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" event={"ID":"d569026a-daa3-47c4-b080-6c4ad7d3f31a","Type":"ContainerStarted","Data":"5e4a0b592a64c0b14eb30ec81ba2f6592f085cf945ca846eac0eba0c9c50d192"}
Sep 09 21:06:13.804213 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:13.804152    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" podStartSLOduration=1.8041374000000001 podStartE2EDuration="1.8041374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:13.802823329 +0000 UTC m=+1050.457603487" watchObservedRunningTime="2026-09-09 21:06:13.8041374 +0000 UTC m=+1050.458917557"
Sep 09 21:06:13.986004 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:13.985974    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:14.719168 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:14.719127    2578 patch_prober.go:28] interesting pod/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 21:06:14.719358 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:14.719209    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:14.991314 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:14.991243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:15.789985 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:15.789940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:15.791306 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:15.791284    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:17.186098 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.186070    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:06:17.239622 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.239598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8hht\" (UniqueName: \"kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-kube-api-access-g8hht\") pod \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") "
Sep 09 21:06:17.239751 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.239654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-certs\") pod \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") "
Sep 09 21:06:17.239751 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.239698    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-podinfo\") pod \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") "
Sep 09 21:06:17.239751 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.239747    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-socket\") pod \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") "
Sep 09 21:06:17.239867 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.239781    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-data\") pod \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") "
Sep 09 21:06:17.239867 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.239805    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-token\") pod \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") "
Sep 09 21:06:17.239867 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.239847    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-credential-socket\") pod \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") "
Sep 09 21:06:17.239867 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.239846    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" (UID: "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:17.240067 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.239906    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-envoy\") pod \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") "
Sep 09 21:06:17.240067 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.239941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istiod-ca-cert\") pod \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\" (UID: \"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e\") "
Sep 09 21:06:17.240189 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.240056    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" (UID: "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:17.240189 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.240123    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" (UID: "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:17.240368 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.240311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" (UID: "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:17.240368 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.240343    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:17.240368 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.240369    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:17.240598 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.240384    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:17.240598 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.240409    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" (UID: "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:17.242069 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.242042    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-kube-api-access-g8hht" (OuterVolumeSpecName: "kube-api-access-g8hht") pod "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" (UID: "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e"). InnerVolumeSpecName "kube-api-access-g8hht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:17.242220 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.242046    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" (UID: "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:06:17.242220 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.242143    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" (UID: "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:17.242353 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.242326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" (UID: "ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:17.341758 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.341726    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:17.341758 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.341754    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:17.341947 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.341766    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:17.341947 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.341779    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:17.341947 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.341790    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:17.341947 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.341800    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8hht\" (UniqueName: \"kubernetes.io/projected/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e-kube-api-access-g8hht\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:17.800352 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.800266    2578 generic.go:356] "Generic (PLEG): container finished" podID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerID="6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d" exitCode=0
Sep 09 21:06:17.800352 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.800321    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"
Sep 09 21:06:17.800535 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.800351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" event={"ID":"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e","Type":"ContainerDied","Data":"6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d"}
Sep 09 21:06:17.800535 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.800407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn" event={"ID":"ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e","Type":"ContainerDied","Data":"704574503eeb5792b7166a4f52450daaee4eab11e8603eeba0ec56213bff535e"}
Sep 09 21:06:17.800535 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.800431    2578 scope.go:117] "RemoveContainer" containerID="6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d"
Sep 09 21:06:17.810330 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.810312    2578 scope.go:117] "RemoveContainer" containerID="f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90"
Sep 09 21:06:17.819431 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.818071    2578 scope.go:117] "RemoveContainer" containerID="6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d"
Sep 09 21:06:17.819431 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:06:17.818396    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d\": container with ID starting with 6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d not found: ID does not exist" containerID="6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d"
Sep 09 21:06:17.819431 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.818424    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d"} err="failed to get container status \"6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d\": rpc error: code = NotFound desc = could not find container \"6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d\": container with ID starting with 6f130affe50313513b94f4c211f3a435c97703a9ac22d5dc9f87c2f2c050600d not found: ID does not exist"
Sep 09 21:06:17.819431 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.818444    2578 scope.go:117] "RemoveContainer" containerID="f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90"
Sep 09 21:06:17.819431 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:06:17.818712    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90\": container with ID starting with f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90 not found: ID does not exist" containerID="f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90"
Sep 09 21:06:17.819431 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.818735    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90"} err="failed to get container status \"f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90\": rpc error: code = NotFound desc = could not find container \"f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90\": container with ID starting with f7daac6601edef3825a344168a554f7f0e67b456513bfcd719edda7cc77b1b90 not found: ID does not exist"
Sep 09 21:06:17.830585 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.830564    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"]
Sep 09 21:06:17.846104 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:17.846081    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e62fbb04-gw-openshift-default-77b86d7bf6-wvsxn"]
Sep 09 21:06:19.845789 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:19.845755    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" path="/var/lib/kubelet/pods/ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e/volumes"
Sep 09 21:06:20.726057 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:20.726019    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:06:31.671080 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:31.671049    2578 patch_prober.go:28] interesting pod/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": read tcp 10.134.0.2:43962->10.134.0.30:15021: read: connection reset by peer" start-of-body=
Sep 09 21:06:31.671415 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:31.671115    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": read tcp 10.134.0.2:43962->10.134.0.30:15021: read: connection reset by peer"
Sep 09 21:06:31.871289 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:31.871216    2578 generic.go:356] "Generic (PLEG): container finished" podID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerID="0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a" exitCode=0
Sep 09 21:06:31.871420 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:31.871290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" event={"ID":"d569026a-daa3-47c4-b080-6c4ad7d3f31a","Type":"ContainerDied","Data":"0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a"}
Sep 09 21:06:31.871717 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:31.871702    2578 scope.go:117] "RemoveContainer" containerID="0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a"
Sep 09 21:06:31.877853 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:31.873262    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:06:31.877853 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:31.873434    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:06:31.877853 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:31.873559    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:06:32.877876 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:32.877837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" event={"ID":"d569026a-daa3-47c4-b080-6c4ad7d3f31a","Type":"ContainerStarted","Data":"77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f"}
Sep 09 21:06:32.987030 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:32.986990    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:32.987244 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:32.987220    2578 patch_prober.go:28] interesting pod/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 09 21:06:32.987321 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:32.987271    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 09 21:06:33.986268 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:33.986233    2578 patch_prober.go:28] interesting pod/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 09 21:06:33.986691 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:33.986295    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 09 21:06:34.986680 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:34.986643    2578 patch_prober.go:28] interesting pod/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 09 21:06:34.987048 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:34.986696    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 09 21:06:35.986625 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:35.986538    2578 patch_prober.go:28] interesting pod/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 09 21:06:35.986625 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:35.986603    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 09 21:06:36.986652 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:36.986617    2578 patch_prober.go:28] interesting pod/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 09 21:06:36.986823 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:36.986683    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 09 21:06:37.986743 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:37.986703    2578 patch_prober.go:28] interesting pod/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 09 21:06:37.986946 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:37.986789    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 09 21:06:38.990382 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:38.990354    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:38.990791 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:38.990771    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:38.991481 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:38.991460    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:39.115411 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:39.115382    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:06:40.844447 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:40.844415    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"]
Sep 09 21:06:40.844943 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:40.844854    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy"
Sep 09 21:06:40.844943 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:40.844873    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy"
Sep 09 21:06:40.845070 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:40.844977    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy"
Sep 09 21:06:40.847020 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:40.846999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"
Sep 09 21:06:40.849276 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:40.849252    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6l2wq\""
Sep 09 21:06:40.855889 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:40.855867    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"]
Sep 09 21:06:40.977009 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:40.976976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4867cc5c-12f2-4ad5-9f2e-ed133a143bf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs\" (UID: \"4867cc5c-12f2-4ad5-9f2e-ed133a143bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"
Sep 09 21:06:41.078108 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:41.078071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4867cc5c-12f2-4ad5-9f2e-ed133a143bf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs\" (UID: \"4867cc5c-12f2-4ad5-9f2e-ed133a143bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"
Sep 09 21:06:41.078679 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:41.078659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4867cc5c-12f2-4ad5-9f2e-ed133a143bf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs\" (UID: \"4867cc5c-12f2-4ad5-9f2e-ed133a143bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"
Sep 09 21:06:41.158224 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:41.158154    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"
Sep 09 21:06:41.424780 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:41.424700    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"]
Sep 09 21:06:41.427595 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:06:41.427570    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4867cc5c_12f2_4ad5_9f2e_ed133a143bf1.slice/crio-7588a45d6575e8f3f16c5bcf7c0b56162d5af6e53274dc1b3183dc9990bae566 WatchSource:0}: Error finding container 7588a45d6575e8f3f16c5bcf7c0b56162d5af6e53274dc1b3183dc9990bae566: Status 404 returned error can't find the container with id 7588a45d6575e8f3f16c5bcf7c0b56162d5af6e53274dc1b3183dc9990bae566
Sep 09 21:06:41.918656 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:41.918623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs" event={"ID":"4867cc5c-12f2-4ad5-9f2e-ed133a143bf1","Type":"ContainerStarted","Data":"940d6d1ce359242516366cc4093c9aaf816215b180719056923a16ffd681348b"}
Sep 09 21:06:41.919023 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:41.918674    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs" event={"ID":"4867cc5c-12f2-4ad5-9f2e-ed133a143bf1","Type":"ContainerStarted","Data":"7588a45d6575e8f3f16c5bcf7c0b56162d5af6e53274dc1b3183dc9990bae566"}
Sep 09 21:06:41.933773 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:41.933726    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs" podStartSLOduration=1.933710483 podStartE2EDuration="1.933710483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:41.932218305 +0000 UTC m=+1078.586998463" watchObservedRunningTime="2026-09-09 21:06:41.933710483 +0000 UTC m=+1078.588490640"
Sep 09 21:06:42.924037 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:42.923999    2578 generic.go:356] "Generic (PLEG): container finished" podID="4867cc5c-12f2-4ad5-9f2e-ed133a143bf1" containerID="940d6d1ce359242516366cc4093c9aaf816215b180719056923a16ffd681348b" exitCode=0
Sep 09 21:06:42.924420 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:42.924050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs" event={"ID":"4867cc5c-12f2-4ad5-9f2e-ed133a143bf1","Type":"ContainerDied","Data":"940d6d1ce359242516366cc4093c9aaf816215b180719056923a16ffd681348b"}
Sep 09 21:06:44.052783 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:44.052728    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"
Sep 09 21:06:44.204406 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:44.204372    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4867cc5c-12f2-4ad5-9f2e-ed133a143bf1-maas-api-service-ca\") pod \"4867cc5c-12f2-4ad5-9f2e-ed133a143bf1\" (UID: \"4867cc5c-12f2-4ad5-9f2e-ed133a143bf1\") "
Sep 09 21:06:44.204717 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:44.204694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4867cc5c-12f2-4ad5-9f2e-ed133a143bf1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4867cc5c-12f2-4ad5-9f2e-ed133a143bf1" (UID: "4867cc5c-12f2-4ad5-9f2e-ed133a143bf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:44.305214 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:44.305124    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4867cc5c-12f2-4ad5-9f2e-ed133a143bf1-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:44.934044 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:44.934010    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"
Sep 09 21:06:44.934044 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:44.934033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs" event={"ID":"4867cc5c-12f2-4ad5-9f2e-ed133a143bf1","Type":"ContainerDied","Data":"7588a45d6575e8f3f16c5bcf7c0b56162d5af6e53274dc1b3183dc9990bae566"}
Sep 09 21:06:44.934294 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:44.934073    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7588a45d6575e8f3f16c5bcf7c0b56162d5af6e53274dc1b3183dc9990bae566"
Sep 09 21:06:45.306206 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.306112    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"]
Sep 09 21:06:45.306624 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.306565    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy"
Sep 09 21:06:45.306624 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.306586    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy"
Sep 09 21:06:45.306624 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.306607    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4867cc5c-12f2-4ad5-9f2e-ed133a143bf1" containerName="revoke-keys"
Sep 09 21:06:45.306624 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.306615    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4867cc5c-12f2-4ad5-9f2e-ed133a143bf1" containerName="revoke-keys"
Sep 09 21:06:45.306825 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.306706    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4867cc5c-12f2-4ad5-9f2e-ed133a143bf1" containerName="revoke-keys"
Sep 09 21:06:45.306825 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.306721    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3de8a3-4e99-4a25-a2e0-58ca9ed1919e" containerName="istio-proxy"
Sep 09 21:06:45.308630 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.308610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.311185 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.311151    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9a54ea\"/\"openshift-service-ca.crt\""
Sep 09 21:06:45.311663 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.311637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9a54ea\"/\"default-dockercfg-4wjqs\""
Sep 09 21:06:45.311791 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.311699    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9a54ea\"/\"kube-root-ca.crt\""
Sep 09 21:06:45.311791 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.311705    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9a54ea\"/\"test-model-9a54ea-kserve-self-signed-certs\""
Sep 09 21:06:45.318765 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.318735    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"]
Sep 09 21:06:45.415069 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.415030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df08a066-11dc-43df-8245-c9906f780c93-tls-certs\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.415258 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.415076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-home\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.415258 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.415102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-dshm\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.415258 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.415167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-model-cache\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.415368 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.415282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-tmp-dir\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.415368 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.415338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2584f\" (UniqueName: \"kubernetes.io/projected/df08a066-11dc-43df-8245-c9906f780c93-kube-api-access-2584f\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.516212 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.516156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-home\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.516212 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.516214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-dshm\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.516429 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.516245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-model-cache\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.516429 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.516289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-tmp-dir\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.516429 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.516343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2584f\" (UniqueName: \"kubernetes.io/projected/df08a066-11dc-43df-8245-c9906f780c93-kube-api-access-2584f\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.516429 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.516385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df08a066-11dc-43df-8245-c9906f780c93-tls-certs\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.516640 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.516593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-home\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.516789 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.516766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-model-cache\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.517085 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.517026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-tmp-dir\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.519656 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.519522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-dshm\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.519774 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.519695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df08a066-11dc-43df-8245-c9906f780c93-tls-certs\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.525527 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.525479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2584f\" (UniqueName: \"kubernetes.io/projected/df08a066-11dc-43df-8245-c9906f780c93-kube-api-access-2584f\") pod \"test-model-9a54ea-kserve-797f76d4f9-s58b5\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") " pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.619864 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.619839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:45.903232 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.903149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"]
Sep 09 21:06:45.905986 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:06:45.905960    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf08a066_11dc_43df_8245_c9906f780c93.slice/crio-8bfae3fc78e32c9c41c89ca55182028c9069574e44ceda7789e879c9bd29af99 WatchSource:0}: Error finding container 8bfae3fc78e32c9c41c89ca55182028c9069574e44ceda7789e879c9bd29af99: Status 404 returned error can't find the container with id 8bfae3fc78e32c9c41c89ca55182028c9069574e44ceda7789e879c9bd29af99
Sep 09 21:06:45.939946 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:45.939924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5" event={"ID":"df08a066-11dc-43df-8245-c9906f780c93","Type":"ContainerStarted","Data":"8bfae3fc78e32c9c41c89ca55182028c9069574e44ceda7789e879c9bd29af99"}
Sep 09 21:06:46.946034 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:46.945994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5" event={"ID":"df08a066-11dc-43df-8245-c9906f780c93","Type":"ContainerStarted","Data":"495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f"}
Sep 09 21:06:46.963951 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:46.963904    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5" podStartSLOduration=1.9638903060000001 podStartE2EDuration="1.963890306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:46.96306132 +0000 UTC m=+1083.617841479" watchObservedRunningTime="2026-09-09 21:06:46.963890306 +0000 UTC m=+1083.618670495"
Sep 09 21:06:47.467921 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:47.467886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"]
Sep 09 21:06:47.471281 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:47.471255    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-857dd97d-vhzjs"]
Sep 09 21:06:47.840631 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:47.840602    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4867cc5c-12f2-4ad5-9f2e-ed133a143bf1" path="/var/lib/kubelet/pods/4867cc5c-12f2-4ad5-9f2e-ed133a143bf1/volumes"
Sep 09 21:06:48.564769 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:48.564737    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"]
Sep 09 21:06:48.565163 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:48.564933    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy" containerID="cri-o://77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f" gracePeriod=30
Sep 09 21:06:48.803552 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:48.803517    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"]
Sep 09 21:06:49.892537 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:49.892508    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"]
Sep 09 21:06:53.711460 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.711379    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"]
Sep 09 21:06:53.714651 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.714627    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.717249 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.717233    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-47d82e7b-serving-cert\""
Sep 09 21:06:53.727244 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.727221    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"]
Sep 09 21:06:53.789356 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.789327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-openshift-service-ca\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.789493 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.789382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-maas-api-tls\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.789493 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.789461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-metrics-certs\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.789568 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.789513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zqgn\" (UniqueName: \"kubernetes.io/projected/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-kube-api-access-4zqgn\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.828887 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.828865    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:53.890523 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890450    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-envoy\") pod \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") "
Sep 09 21:06:53.890523 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zwjr\" (UniqueName: \"kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-kube-api-access-8zwjr\") pod \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") "
Sep 09 21:06:53.890523 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890514    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-data\") pod \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") "
Sep 09 21:06:53.890774 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890532    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-socket\") pod \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") "
Sep 09 21:06:53.890774 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890639    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-podinfo\") pod \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") "
Sep 09 21:06:53.890774 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890712    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istiod-ca-cert\") pod \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") "
Sep 09 21:06:53.890774 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890740    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-certs\") pod \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") "
Sep 09 21:06:53.890965 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890773    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-token\") pod \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") "
Sep 09 21:06:53.890965 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890798    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-credential-socket\") pod \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\" (UID: \"d569026a-daa3-47c4-b080-6c4ad7d3f31a\") "
Sep 09 21:06:53.890965 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d569026a-daa3-47c4-b080-6c4ad7d3f31a" (UID: "d569026a-daa3-47c4-b080-6c4ad7d3f31a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:53.890965 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890849    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-data" (OuterVolumeSpecName: "istio-data") pod "d569026a-daa3-47c4-b080-6c4ad7d3f31a" (UID: "d569026a-daa3-47c4-b080-6c4ad7d3f31a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:53.890965 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.890916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-maas-api-tls\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.891238 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.891022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-metrics-certs\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.891238 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.891108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqgn\" (UniqueName: \"kubernetes.io/projected/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-kube-api-access-4zqgn\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.891238 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.891140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-openshift-service-ca\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.891238 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.891229    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:53.891435 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.891246    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:53.891435 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.891014    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d569026a-daa3-47c4-b080-6c4ad7d3f31a" (UID: "d569026a-daa3-47c4-b080-6c4ad7d3f31a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:53.891435 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.891222    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d569026a-daa3-47c4-b080-6c4ad7d3f31a" (UID: "d569026a-daa3-47c4-b080-6c4ad7d3f31a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:53.891435 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:06:53.891224    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-serving-cert: secret "maas-api-e2e-derive-47d82e7b-serving-cert" not found
Sep 09 21:06:53.891435 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:06:53.891332    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-maas-api-tls podName:1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c nodeName:}" failed. No retries permitted until 2026-09-09 21:06:54.391311591 +0000 UTC m=+1091.046091729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-maas-api-tls") pod "maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g" (UID: "1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c") : secret "maas-api-e2e-derive-47d82e7b-serving-cert" not found
Sep 09 21:06:53.891693 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.891251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d569026a-daa3-47c4-b080-6c4ad7d3f31a" (UID: "d569026a-daa3-47c4-b080-6c4ad7d3f31a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:53.892017 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.891989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-openshift-service-ca\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.893123 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.893100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-token" (OuterVolumeSpecName: "istio-token") pod "d569026a-daa3-47c4-b080-6c4ad7d3f31a" (UID: "d569026a-daa3-47c4-b080-6c4ad7d3f31a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:53.893292 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.893274    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-kube-api-access-8zwjr" (OuterVolumeSpecName: "kube-api-access-8zwjr") pod "d569026a-daa3-47c4-b080-6c4ad7d3f31a" (UID: "d569026a-daa3-47c4-b080-6c4ad7d3f31a"). InnerVolumeSpecName "kube-api-access-8zwjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:53.893468 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.893443    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d569026a-daa3-47c4-b080-6c4ad7d3f31a" (UID: "d569026a-daa3-47c4-b080-6c4ad7d3f31a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:06:53.893691 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.893664    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d569026a-daa3-47c4-b080-6c4ad7d3f31a" (UID: "d569026a-daa3-47c4-b080-6c4ad7d3f31a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:53.893974 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.893957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-metrics-certs\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.901251 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.901207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zqgn\" (UniqueName: \"kubernetes.io/projected/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-kube-api-access-4zqgn\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:53.978813 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.978785    2578 generic.go:356] "Generic (PLEG): container finished" podID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerID="77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f" exitCode=0
Sep 09 21:06:53.978947 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.978840    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"
Sep 09 21:06:53.978947 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.978861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" event={"ID":"d569026a-daa3-47c4-b080-6c4ad7d3f31a","Type":"ContainerDied","Data":"77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f"}
Sep 09 21:06:53.978947 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.978911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv" event={"ID":"d569026a-daa3-47c4-b080-6c4ad7d3f31a","Type":"ContainerDied","Data":"5e4a0b592a64c0b14eb30ec81ba2f6592f085cf945ca846eac0eba0c9c50d192"}
Sep 09 21:06:53.978947 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.978933    2578 scope.go:117] "RemoveContainer" containerID="77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f"
Sep 09 21:06:53.987959 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.987935    2578 scope.go:117] "RemoveContainer" containerID="0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a"
Sep 09 21:06:53.991810 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.991791    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:53.991911 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.991815    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zwjr\" (UniqueName: \"kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-kube-api-access-8zwjr\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:53.991911 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.991832    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:53.991911 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.991847    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:53.991911 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.991861    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:53.991911 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.991874    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d569026a-daa3-47c4-b080-6c4ad7d3f31a-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:53.991911 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.991887    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d569026a-daa3-47c4-b080-6c4ad7d3f31a-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:06:53.998604 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.998589    2578 scope.go:117] "RemoveContainer" containerID="77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f"
Sep 09 21:06:53.998898 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:06:53.998876    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f\": container with ID starting with 77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f not found: ID does not exist" containerID="77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f"
Sep 09 21:06:53.998963 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.998905    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f"} err="failed to get container status \"77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f\": rpc error: code = NotFound desc = could not find container \"77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f\": container with ID starting with 77edca3999d30a89c846bbc0c10a5c48e28bb9b2d33f26d0d4db45074e47d12f not found: ID does not exist"
Sep 09 21:06:53.998963 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.998921    2578 scope.go:117] "RemoveContainer" containerID="0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a"
Sep 09 21:06:53.999125 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:06:53.999106    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a\": container with ID starting with 0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a not found: ID does not exist" containerID="0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a"
Sep 09 21:06:53.999198 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:53.999130    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a"} err="failed to get container status \"0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a\": rpc error: code = NotFound desc = could not find container \"0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a\": container with ID starting with 0b04aafd31079f4d493ee24f82a10e2c18e1dfeac5e82c337ac2ed9ea6924f1a not found: ID does not exist"
Sep 09 21:06:54.006381 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:54.006361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"]
Sep 09 21:06:54.011695 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:54.011676    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-857dd97d-openshift-default-7b74fff956-2ddjv"]
Sep 09 21:06:54.394057 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:54.394013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-maas-api-tls\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:54.396644 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:54.396619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-maas-api-tls\") pod \"maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:54.627159 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:54.627126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:54.898264 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:54.898232    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"]
Sep 09 21:06:54.900913 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:06:54.900884    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f49d6b6_2d57_41b5_ad5b_5b6cf0815d5c.slice/crio-ccd35f4ac84235c574dd6b1092338e64a08ecd68f9dd1557eb42e9d7c653b953 WatchSource:0}: Error finding container ccd35f4ac84235c574dd6b1092338e64a08ecd68f9dd1557eb42e9d7c653b953: Status 404 returned error can't find the container with id ccd35f4ac84235c574dd6b1092338e64a08ecd68f9dd1557eb42e9d7c653b953
Sep 09 21:06:54.984995 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:54.984919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g" event={"ID":"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c","Type":"ContainerStarted","Data":"ccd35f4ac84235c574dd6b1092338e64a08ecd68f9dd1557eb42e9d7c653b953"}
Sep 09 21:06:55.620076 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:55.620037    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:55.620262 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:55.620090    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:55.632313 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:55.632287    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:55.841661 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:55.841629    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" path="/var/lib/kubelet/pods/d569026a-daa3-47c4-b080-6c4ad7d3f31a/volumes"
Sep 09 21:06:55.991766 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:55.991677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g" event={"ID":"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c","Type":"ContainerStarted","Data":"f17c3b56670bbb47c12b43407792b702ecbd07274ef6a6cfa21787e2ee22f40e"}
Sep 09 21:06:55.992126 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:55.991988    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:06:56.003696 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:56.003676    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:06:56.010092 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:06:56.010051    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g" podStartSLOduration=2.14548812 podStartE2EDuration="3.010037118s" totalImagesPullingTime="864.548998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:56.008737625 +0000 UTC m=+1092.663517784" watchObservedRunningTime="2026-09-09 21:06:56.010037118 +0000 UTC m=+1092.664817269"
Sep 09 21:07:02.001929 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:02.001897    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:07:02.946775 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:02.946742    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"]
Sep 09 21:07:02.946979 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:02.946955    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch" podUID="ef7589cd-3cf1-4b2c-b413-8f44ba6060ed" containerName="maas-api" containerID="cri-o://cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a" gracePeriod=30
Sep 09 21:07:03.190231 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.190206    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:07:03.274323 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.274239    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-openshift-service-ca\") pod \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") "
Sep 09 21:07:03.274323 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.274273    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-metrics-certs\") pod \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") "
Sep 09 21:07:03.274323 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.274300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-maas-api-tls\") pod \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") "
Sep 09 21:07:03.274600 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.274334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgs7r\" (UniqueName: \"kubernetes.io/projected/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-kube-api-access-kgs7r\") pod \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\" (UID: \"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed\") "
Sep 09 21:07:03.274685 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.274659    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef7589cd-3cf1-4b2c-b413-8f44ba6060ed" (UID: "ef7589cd-3cf1-4b2c-b413-8f44ba6060ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:03.276514 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.276483    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ef7589cd-3cf1-4b2c-b413-8f44ba6060ed" (UID: "ef7589cd-3cf1-4b2c-b413-8f44ba6060ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:03.276615 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.276530    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef7589cd-3cf1-4b2c-b413-8f44ba6060ed" (UID: "ef7589cd-3cf1-4b2c-b413-8f44ba6060ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:03.276615 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.276546    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-kube-api-access-kgs7r" (OuterVolumeSpecName: "kube-api-access-kgs7r") pod "ef7589cd-3cf1-4b2c-b413-8f44ba6060ed" (UID: "ef7589cd-3cf1-4b2c-b413-8f44ba6060ed"). InnerVolumeSpecName "kube-api-access-kgs7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:03.375532 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.375496    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:03.375532 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.375526    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:03.375532 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.375535    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:03.375783 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.375544    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgs7r\" (UniqueName: \"kubernetes.io/projected/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed-kube-api-access-kgs7r\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:03.822902 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:03.822863    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7" podUID="c64af80e-2d08-468b-824a-4aa2f4b57510" containerName="main" containerID="cri-o://637bf9838ab3a49583d3ce4226f68f3c9a85dbfb82f0c08b8d6cf75089b9b13e" gracePeriod=45
Sep 09 21:07:04.029557 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.029518    2578 generic.go:356] "Generic (PLEG): container finished" podID="ef7589cd-3cf1-4b2c-b413-8f44ba6060ed" containerID="cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a" exitCode=0
Sep 09 21:07:04.029743 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.029578    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"
Sep 09 21:07:04.029743 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.029599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch" event={"ID":"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed","Type":"ContainerDied","Data":"cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a"}
Sep 09 21:07:04.029743 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.029693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch" event={"ID":"ef7589cd-3cf1-4b2c-b413-8f44ba6060ed","Type":"ContainerDied","Data":"ec3e5edd63ff1439fe1a79551fd53f7a704296ae739d2a06a4a8573f439969b1"}
Sep 09 21:07:04.029743 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.029724    2578 scope.go:117] "RemoveContainer" containerID="cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a"
Sep 09 21:07:04.031653 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.031628    2578 generic.go:356] "Generic (PLEG): container finished" podID="c64af80e-2d08-468b-824a-4aa2f4b57510" containerID="637bf9838ab3a49583d3ce4226f68f3c9a85dbfb82f0c08b8d6cf75089b9b13e" exitCode=0
Sep 09 21:07:04.031787 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.031734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7" event={"ID":"c64af80e-2d08-468b-824a-4aa2f4b57510","Type":"ContainerDied","Data":"637bf9838ab3a49583d3ce4226f68f3c9a85dbfb82f0c08b8d6cf75089b9b13e"}
Sep 09 21:07:04.038316 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.038296    2578 scope.go:117] "RemoveContainer" containerID="cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a"
Sep 09 21:07:04.038596 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:07:04.038577    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a\": container with ID starting with cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a not found: ID does not exist" containerID="cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a"
Sep 09 21:07:04.038653 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.038603    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a"} err="failed to get container status \"cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a\": rpc error: code = NotFound desc = could not find container \"cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a\": container with ID starting with cb0e7e5e9eebc3db863d85d93e58d603e0ef1024247db5fb39c040f28f63568a not found: ID does not exist"
Sep 09 21:07:04.046333 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.046311    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"]
Sep 09 21:07:04.050787 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.050767    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80db9b-86679c6bb7-84xch"]
Sep 09 21:07:04.065901 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.065882    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:07:04.182713 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.182680    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5zqp\" (UniqueName: \"kubernetes.io/projected/c64af80e-2d08-468b-824a-4aa2f4b57510-kube-api-access-j5zqp\") pod \"c64af80e-2d08-468b-824a-4aa2f4b57510\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") "
Sep 09 21:07:04.182875 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.182728    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64af80e-2d08-468b-824a-4aa2f4b57510-tls-certs\") pod \"c64af80e-2d08-468b-824a-4aa2f4b57510\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") "
Sep 09 21:07:04.182875 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.182778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-tmp-dir\") pod \"c64af80e-2d08-468b-824a-4aa2f4b57510\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") "
Sep 09 21:07:04.182875 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.182816    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-home\") pod \"c64af80e-2d08-468b-824a-4aa2f4b57510\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") "
Sep 09 21:07:04.182875 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.182863    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-model-cache\") pod \"c64af80e-2d08-468b-824a-4aa2f4b57510\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") "
Sep 09 21:07:04.183054 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.182910    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-dshm\") pod \"c64af80e-2d08-468b-824a-4aa2f4b57510\" (UID: \"c64af80e-2d08-468b-824a-4aa2f4b57510\") "
Sep 09 21:07:04.183136 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.183102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c64af80e-2d08-468b-824a-4aa2f4b57510" (UID: "c64af80e-2d08-468b-824a-4aa2f4b57510"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:04.183227 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.183122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-home" (OuterVolumeSpecName: "home") pod "c64af80e-2d08-468b-824a-4aa2f4b57510" (UID: "c64af80e-2d08-468b-824a-4aa2f4b57510"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:04.183227 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.183164    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-model-cache" (OuterVolumeSpecName: "model-cache") pod "c64af80e-2d08-468b-824a-4aa2f4b57510" (UID: "c64af80e-2d08-468b-824a-4aa2f4b57510"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:04.183345 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.183271    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-tmp-dir\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:04.183345 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.183290    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-home\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:04.185245 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.185221    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64af80e-2d08-468b-824a-4aa2f4b57510-kube-api-access-j5zqp" (OuterVolumeSpecName: "kube-api-access-j5zqp") pod "c64af80e-2d08-468b-824a-4aa2f4b57510" (UID: "c64af80e-2d08-468b-824a-4aa2f4b57510"). InnerVolumeSpecName "kube-api-access-j5zqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:04.185364 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.185306    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64af80e-2d08-468b-824a-4aa2f4b57510-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c64af80e-2d08-468b-824a-4aa2f4b57510" (UID: "c64af80e-2d08-468b-824a-4aa2f4b57510"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:04.185421 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.185391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-dshm" (OuterVolumeSpecName: "dshm") pod "c64af80e-2d08-468b-824a-4aa2f4b57510" (UID: "c64af80e-2d08-468b-824a-4aa2f4b57510"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:04.206930 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.206900    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:07:04.284561 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.284525    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-model-cache\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:04.284561 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.284557    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64af80e-2d08-468b-824a-4aa2f4b57510-dshm\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:04.284561 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.284567    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5zqp\" (UniqueName: \"kubernetes.io/projected/c64af80e-2d08-468b-824a-4aa2f4b57510-kube-api-access-j5zqp\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:04.284943 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.284582    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64af80e-2d08-468b-824a-4aa2f4b57510-tls-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:04.911374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:04.911333    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v" podUID="ac6f15f4-1a14-4423-a1dc-060a01698302" containerName="main" containerID="cri-o://3072bc1ebb790bc39e0e95a894105f46635393cc6d26ff64e0ca5b9ed3e05796" gracePeriod=45
Sep 09 21:07:05.038919 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.038891    2578 generic.go:356] "Generic (PLEG): container finished" podID="ac6f15f4-1a14-4423-a1dc-060a01698302" containerID="3072bc1ebb790bc39e0e95a894105f46635393cc6d26ff64e0ca5b9ed3e05796" exitCode=0
Sep 09 21:07:05.039047 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.038963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v" event={"ID":"ac6f15f4-1a14-4423-a1dc-060a01698302","Type":"ContainerDied","Data":"3072bc1ebb790bc39e0e95a894105f46635393cc6d26ff64e0ca5b9ed3e05796"}
Sep 09 21:07:05.041424 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.041402    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"
Sep 09 21:07:05.041536 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.041405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7" event={"ID":"c64af80e-2d08-468b-824a-4aa2f4b57510","Type":"ContainerDied","Data":"39822e3ece869de27b29bfe48d65d9dafeeb827f0324b3c55ba54184fe4b06d8"}
Sep 09 21:07:05.041536 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.041521    2578 scope.go:117] "RemoveContainer" containerID="637bf9838ab3a49583d3ce4226f68f3c9a85dbfb82f0c08b8d6cf75089b9b13e"
Sep 09 21:07:05.062538 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.062514    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"]
Sep 09 21:07:05.064719 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.064695    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-5927d1b0-kserve-864d9bf785-xxmr7"]
Sep 09 21:07:05.152463 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.152442    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:07:05.292983 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.292890    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-tmp-dir\") pod \"ac6f15f4-1a14-4423-a1dc-060a01698302\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") "
Sep 09 21:07:05.292983 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.292945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcpmr\" (UniqueName: \"kubernetes.io/projected/ac6f15f4-1a14-4423-a1dc-060a01698302-kube-api-access-lcpmr\") pod \"ac6f15f4-1a14-4423-a1dc-060a01698302\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") "
Sep 09 21:07:05.293450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.292984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-home\") pod \"ac6f15f4-1a14-4423-a1dc-060a01698302\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") "
Sep 09 21:07:05.293450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.293016    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-model-cache\") pod \"ac6f15f4-1a14-4423-a1dc-060a01698302\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") "
Sep 09 21:07:05.293450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.293043    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-dshm\") pod \"ac6f15f4-1a14-4423-a1dc-060a01698302\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") "
Sep 09 21:07:05.293450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.293096    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6f15f4-1a14-4423-a1dc-060a01698302-tls-certs\") pod \"ac6f15f4-1a14-4423-a1dc-060a01698302\" (UID: \"ac6f15f4-1a14-4423-a1dc-060a01698302\") "
Sep 09 21:07:05.293450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.293195    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac6f15f4-1a14-4423-a1dc-060a01698302" (UID: "ac6f15f4-1a14-4423-a1dc-060a01698302"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.293450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.293284    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac6f15f4-1a14-4423-a1dc-060a01698302" (UID: "ac6f15f4-1a14-4423-a1dc-060a01698302"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.293450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.293301    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-home" (OuterVolumeSpecName: "home") pod "ac6f15f4-1a14-4423-a1dc-060a01698302" (UID: "ac6f15f4-1a14-4423-a1dc-060a01698302"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.293450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.293434    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-tmp-dir\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.293450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.293448    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-home\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.293450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.293457    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-model-cache\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.295137 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.295107    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-dshm" (OuterVolumeSpecName: "dshm") pod "ac6f15f4-1a14-4423-a1dc-060a01698302" (UID: "ac6f15f4-1a14-4423-a1dc-060a01698302"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.295260 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.295147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6f15f4-1a14-4423-a1dc-060a01698302-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac6f15f4-1a14-4423-a1dc-060a01698302" (UID: "ac6f15f4-1a14-4423-a1dc-060a01698302"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:05.295260 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.295160    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6f15f4-1a14-4423-a1dc-060a01698302-kube-api-access-lcpmr" (OuterVolumeSpecName: "kube-api-access-lcpmr") pod "ac6f15f4-1a14-4423-a1dc-060a01698302" (UID: "ac6f15f4-1a14-4423-a1dc-060a01698302"). InnerVolumeSpecName "kube-api-access-lcpmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:05.394522 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.394488    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcpmr\" (UniqueName: \"kubernetes.io/projected/ac6f15f4-1a14-4423-a1dc-060a01698302-kube-api-access-lcpmr\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.394522 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.394514    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6f15f4-1a14-4423-a1dc-060a01698302-dshm\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.394522 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.394524    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6f15f4-1a14-4423-a1dc-060a01698302-tls-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.840166 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.840135    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c64af80e-2d08-468b-824a-4aa2f4b57510" path="/var/lib/kubelet/pods/c64af80e-2d08-468b-824a-4aa2f4b57510/volumes"
Sep 09 21:07:05.840571 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:05.840556    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7589cd-3cf1-4b2c-b413-8f44ba6060ed" path="/var/lib/kubelet/pods/ef7589cd-3cf1-4b2c-b413-8f44ba6060ed/volumes"
Sep 09 21:07:06.047769 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:06.047739    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"
Sep 09 21:07:06.047769 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:06.047751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v" event={"ID":"ac6f15f4-1a14-4423-a1dc-060a01698302","Type":"ContainerDied","Data":"53fff6ec198a10f48b2b9c44d880feeaf3818488cec765f0e80ef284ca5025d9"}
Sep 09 21:07:06.047989 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:06.047801    2578 scope.go:117] "RemoveContainer" containerID="3072bc1ebb790bc39e0e95a894105f46635393cc6d26ff64e0ca5b9ed3e05796"
Sep 09 21:07:06.062589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:06.062558    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"]
Sep 09 21:07:06.064333 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:06.064311    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-5927d1b0-kserve-5bc77858c9-zjf9v"]
Sep 09 21:07:07.840425 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:07.840391    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac6f15f4-1a14-4423-a1dc-060a01698302" path="/var/lib/kubelet/pods/ac6f15f4-1a14-4423-a1dc-060a01698302/volumes"
Sep 09 21:07:10.822935 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.822893    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"]
Sep 09 21:07:10.823394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823268    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c64af80e-2d08-468b-824a-4aa2f4b57510" containerName="main"
Sep 09 21:07:10.823394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823286    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c64af80e-2d08-468b-824a-4aa2f4b57510" containerName="main"
Sep 09 21:07:10.823394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823302    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6f15f4-1a14-4423-a1dc-060a01698302" containerName="main"
Sep 09 21:07:10.823394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823310    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6f15f4-1a14-4423-a1dc-060a01698302" containerName="main"
Sep 09 21:07:10.823394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823323    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy"
Sep 09 21:07:10.823394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823332    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy"
Sep 09 21:07:10.823394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823346    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7589cd-3cf1-4b2c-b413-8f44ba6060ed" containerName="maas-api"
Sep 09 21:07:10.823394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823353    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7589cd-3cf1-4b2c-b413-8f44ba6060ed" containerName="maas-api"
Sep 09 21:07:10.823672 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823435    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7589cd-3cf1-4b2c-b413-8f44ba6060ed" containerName="maas-api"
Sep 09 21:07:10.823672 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823445    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6f15f4-1a14-4423-a1dc-060a01698302" containerName="main"
Sep 09 21:07:10.823672 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823454    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c64af80e-2d08-468b-824a-4aa2f4b57510" containerName="main"
Sep 09 21:07:10.823672 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823461    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy"
Sep 09 21:07:10.823672 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.823469    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy"
Sep 09 21:07:10.826269 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.826243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:10.828624 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.828605    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-0066f8-openshift-default-dockercfg-qjlq6\""
Sep 09 21:07:10.838243 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.838219    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"]
Sep 09 21:07:10.944112 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.944070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-envoy\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:10.944112 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.944119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-credential-socket\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:10.944395 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.944199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-istio-token\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:10.944395 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.944253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22865335-cc68-4c31-87e1-de0d0e837f74-istio-podinfo\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:10.944395 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.944312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-certs\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:10.944395 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.944344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-data\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:10.944395 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.944361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45ffn\" (UniqueName: \"kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-kube-api-access-45ffn\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:10.944660 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.944419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-socket\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:10.944660 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:10.944539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22865335-cc68-4c31-87e1-de0d0e837f74-istiod-ca-cert\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.045995 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.045956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-credential-socket\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.045995 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-istio-token\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046239 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22865335-cc68-4c31-87e1-de0d0e837f74-istio-podinfo\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046239 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-certs\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046239 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-data\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046239 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45ffn\" (UniqueName: \"kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-kube-api-access-45ffn\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046239 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-socket\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046239 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22865335-cc68-4c31-87e1-de0d0e837f74-istiod-ca-cert\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046540 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-envoy\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046540 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-credential-socket\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046540 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-certs\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046540 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046524    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-socket\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046683 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-data\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.046985 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.046963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22865335-cc68-4c31-87e1-de0d0e837f74-istiod-ca-cert\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.048350 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.048322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-envoy\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.048587 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.048567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22865335-cc68-4c31-87e1-de0d0e837f74-istio-podinfo\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.054054 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.054028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-istio-token\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.054149 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.054124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45ffn\" (UniqueName: \"kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-kube-api-access-45ffn\") pod \"e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") " pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.136346 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.136317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:11.408202 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.408115    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"]
Sep 09 21:07:11.410815 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:07:11.410780    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22865335_cc68_4c31_87e1_de0d0e837f74.slice/crio-12e6a2cb5e5a8dc9576fdddddb8d4aa2e9de65ad73b6273c21637e35351ad361 WatchSource:0}: Error finding container 12e6a2cb5e5a8dc9576fdddddb8d4aa2e9de65ad73b6273c21637e35351ad361: Status 404 returned error can't find the container with id 12e6a2cb5e5a8dc9576fdddddb8d4aa2e9de65ad73b6273c21637e35351ad361
Sep 09 21:07:11.413188 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.413142    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:07:11.413274 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.413228    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:07:11.413274 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:11.413259    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:07:12.075899 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:12.075870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" event={"ID":"22865335-cc68-4c31-87e1-de0d0e837f74","Type":"ContainerStarted","Data":"07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913"}
Sep 09 21:07:12.076243 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:12.075920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" event={"ID":"22865335-cc68-4c31-87e1-de0d0e837f74","Type":"ContainerStarted","Data":"12e6a2cb5e5a8dc9576fdddddb8d4aa2e9de65ad73b6273c21637e35351ad361"}
Sep 09 21:07:12.097797 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:12.097739    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" podStartSLOduration=2.097718732 podStartE2EDuration="2.097718732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:07:12.095024274 +0000 UTC m=+1108.749804445" watchObservedRunningTime="2026-09-09 21:07:12.097718732 +0000 UTC m=+1108.752498892"
Sep 09 21:07:12.136701 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:12.136665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:12.142020 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:12.141996    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:13.080303 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:13.080273    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:13.081337 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:13.081309    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:15.618694 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.618656    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"]
Sep 09 21:07:15.619264 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.619008    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy"
Sep 09 21:07:15.619264 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.619027    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d569026a-daa3-47c4-b080-6c4ad7d3f31a" containerName="istio-proxy"
Sep 09 21:07:15.622195 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.622150    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.624251 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.624227    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-0066f8-serving-cert\""
Sep 09 21:07:15.633397 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.633375    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"]
Sep 09 21:07:15.789392 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.789352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-maas-api-tls\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.789564 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.789411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8450b7b3-85fd-4f11-bea7-7b906a6baad6-openshift-service-ca\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.789564 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.789499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqrj\" (UniqueName: \"kubernetes.io/projected/8450b7b3-85fd-4f11-bea7-7b906a6baad6-kube-api-access-jhqrj\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.789638 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.789567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-metrics-certs\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.890313 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.890225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-maas-api-tls\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.890313 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.890279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8450b7b3-85fd-4f11-bea7-7b906a6baad6-openshift-service-ca\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.890313 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.890313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqrj\" (UniqueName: \"kubernetes.io/projected/8450b7b3-85fd-4f11-bea7-7b906a6baad6-kube-api-access-jhqrj\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.890555 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.890349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-metrics-certs\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.891028 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.891001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8450b7b3-85fd-4f11-bea7-7b906a6baad6-openshift-service-ca\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.892712 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.892691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-maas-api-tls\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.892801 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.892746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-metrics-certs\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.898316 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.898295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqrj\" (UniqueName: \"kubernetes.io/projected/8450b7b3-85fd-4f11-bea7-7b906a6baad6-kube-api-access-jhqrj\") pod \"maas-api-e2e-explicit-0066f8-865ddc554f-gppln\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:15.935164 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:15.935135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:16.205829 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:16.205708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"]
Sep 09 21:07:16.208677 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:07:16.208634    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8450b7b3_85fd_4f11_bea7_7b906a6baad6.slice/crio-aeba6c0c661e322dafca391d3d6f55eb6483e0b2991c95ada2addeebc9fa6b2b WatchSource:0}: Error finding container aeba6c0c661e322dafca391d3d6f55eb6483e0b2991c95ada2addeebc9fa6b2b: Status 404 returned error can't find the container with id aeba6c0c661e322dafca391d3d6f55eb6483e0b2991c95ada2addeebc9fa6b2b
Sep 09 21:07:17.100481 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:17.100448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln" event={"ID":"8450b7b3-85fd-4f11-bea7-7b906a6baad6","Type":"ContainerStarted","Data":"eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf"}
Sep 09 21:07:17.100814 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:17.100507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln" event={"ID":"8450b7b3-85fd-4f11-bea7-7b906a6baad6","Type":"ContainerStarted","Data":"aeba6c0c661e322dafca391d3d6f55eb6483e0b2991c95ada2addeebc9fa6b2b"}
Sep 09 21:07:17.100814 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:17.100596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:17.120787 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:17.120739    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln" podStartSLOduration=1.311264449 podStartE2EDuration="2.120724144s" totalImagesPullingTime="809.459695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:07:17.119003897 +0000 UTC m=+1113.773784054" watchObservedRunningTime="2026-09-09 21:07:17.120724144 +0000 UTC m=+1113.775504340"
Sep 09 21:07:23.110285 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:23.110257    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:07:23.906272 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:23.906235    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"]
Sep 09 21:07:23.906504 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:23.906475    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g" podUID="1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c" containerName="maas-api" containerID="cri-o://f17c3b56670bbb47c12b43407792b702ecbd07274ef6a6cfa21787e2ee22f40e" gracePeriod=30
Sep 09 21:07:24.135832 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.135799    2578 generic.go:356] "Generic (PLEG): container finished" podID="1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c" containerID="f17c3b56670bbb47c12b43407792b702ecbd07274ef6a6cfa21787e2ee22f40e" exitCode=0
Sep 09 21:07:24.136104 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.135868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g" event={"ID":"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c","Type":"ContainerDied","Data":"f17c3b56670bbb47c12b43407792b702ecbd07274ef6a6cfa21787e2ee22f40e"}
Sep 09 21:07:24.136104 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.135921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g" event={"ID":"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c","Type":"ContainerDied","Data":"ccd35f4ac84235c574dd6b1092338e64a08ecd68f9dd1557eb42e9d7c653b953"}
Sep 09 21:07:24.136104 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.135939    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccd35f4ac84235c574dd6b1092338e64a08ecd68f9dd1557eb42e9d7c653b953"
Sep 09 21:07:24.145119 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.145102    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:07:24.264143 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.264071    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-maas-api-tls\") pod \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") "
Sep 09 21:07:24.264143 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.264102    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zqgn\" (UniqueName: \"kubernetes.io/projected/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-kube-api-access-4zqgn\") pod \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") "
Sep 09 21:07:24.264381 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.264153    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-openshift-service-ca\") pod \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") "
Sep 09 21:07:24.264381 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.264298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-metrics-certs\") pod \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\" (UID: \"1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c\") "
Sep 09 21:07:24.264490 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.264475    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c" (UID: "1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:24.264626 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.264612    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:24.266339 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.266315    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c" (UID: "1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:24.266456 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.266351    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-kube-api-access-4zqgn" (OuterVolumeSpecName: "kube-api-access-4zqgn") pod "1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c" (UID: "1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c"). InnerVolumeSpecName "kube-api-access-4zqgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:24.266456 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.266400    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c" (UID: "1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:24.365082 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.365049    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:24.365082 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.365077    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zqgn\" (UniqueName: \"kubernetes.io/projected/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-kube-api-access-4zqgn\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:24.365297 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:24.365090    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:07:25.140868 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:25.140834    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"
Sep 09 21:07:25.163333 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:25.163301    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"]
Sep 09 21:07:25.165183 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:25.165150    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-47d82e7b-869f49d77f-lpw9g"]
Sep 09 21:07:25.840910 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:25.840867    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c" path="/var/lib/kubelet/pods/1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c/volumes"
Sep 09 21:07:27.209936 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.209899    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"]
Sep 09 21:07:27.210344 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.210228    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c" containerName="maas-api"
Sep 09 21:07:27.210344 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.210243    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c" containerName="maas-api"
Sep 09 21:07:27.210344 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.210295    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f49d6b6-2d57-41b5-ad5b-5b6cf0815d5c" containerName="maas-api"
Sep 09 21:07:27.213423 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.213407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.216666 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.216643    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-98dad6\"/\"openshift-service-ca.crt\""
Sep 09 21:07:27.216796 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.216681    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-98dad6\"/\"test-model-98dad6-kserve-self-signed-certs\""
Sep 09 21:07:27.216796 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.216688    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-98dad6\"/\"kube-root-ca.crt\""
Sep 09 21:07:27.216796 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.216739    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-98dad6\"/\"default-dockercfg-8x4vz\""
Sep 09 21:07:27.220862 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.220839    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:07:27.222977 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.222954    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"]
Sep 09 21:07:27.389194 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.389146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-model-cache\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.389374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.389212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tmp-dir\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.389374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.389234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tls-certs\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.389374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.389263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-dshm\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.389374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.389329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-home\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.389535 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.389394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4dp9\" (UniqueName: \"kubernetes.io/projected/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-kube-api-access-g4dp9\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.489865 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.489777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tls-certs\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.489865 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.489827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-dshm\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.489865 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.489865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-home\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.490117 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.489885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4dp9\" (UniqueName: \"kubernetes.io/projected/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-kube-api-access-g4dp9\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.490117 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.489941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-model-cache\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.490117 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.489970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tmp-dir\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.490406 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.490379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-home\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.490475 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.490386    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tmp-dir\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.490475 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.490429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-model-cache\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.492100 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.492079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-dshm\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.492500 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.492481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tls-certs\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.498398 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.498368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4dp9\" (UniqueName: \"kubernetes.io/projected/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-kube-api-access-g4dp9\") pod \"test-model-98dad6-kserve-7795bc85-pml29\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") " pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.523826 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.523801    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:27.815059 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:27.815000    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"]
Sep 09 21:07:27.817130 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:07:27.817103    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20a51d8c_e4d7_4024_95ea_61d42d8ddebe.slice/crio-dd4c6622cbb6af6aa6188c8749fbdbdd90c82097e8372927c611b6db9674b739 WatchSource:0}: Error finding container dd4c6622cbb6af6aa6188c8749fbdbdd90c82097e8372927c611b6db9674b739: Status 404 returned error can't find the container with id dd4c6622cbb6af6aa6188c8749fbdbdd90c82097e8372927c611b6db9674b739
Sep 09 21:07:28.158339 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:28.158288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29" event={"ID":"20a51d8c-e4d7-4024-95ea-61d42d8ddebe","Type":"ContainerStarted","Data":"0866524d079243d578fafd0b740636c3d2e8b93e4d972af6195be696a5ea133e"}
Sep 09 21:07:28.158534 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:28.158346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29" event={"ID":"20a51d8c-e4d7-4024-95ea-61d42d8ddebe","Type":"ContainerStarted","Data":"dd4c6622cbb6af6aa6188c8749fbdbdd90c82097e8372927c611b6db9674b739"}
Sep 09 21:07:28.174448 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:28.174396    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29" podStartSLOduration=1.17437853 podStartE2EDuration="1.17437853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:07:28.173884216 +0000 UTC m=+1124.828664373" watchObservedRunningTime="2026-09-09 21:07:28.17437853 +0000 UTC m=+1124.829158689"
Sep 09 21:07:37.204394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:37.204296    2578 generic.go:356] "Generic (PLEG): container finished" podID="22865335-cc68-4c31-87e1-de0d0e837f74" containerID="07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913" exitCode=0
Sep 09 21:07:37.204394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:37.204372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" event={"ID":"22865335-cc68-4c31-87e1-de0d0e837f74","Type":"ContainerDied","Data":"07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913"}
Sep 09 21:07:37.204940 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:37.204797    2578 scope.go:117] "RemoveContainer" containerID="07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913"
Sep 09 21:07:37.205847 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:37.205819    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:07:37.205919 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:37.205885    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:07:37.205977 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:37.205916    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:07:37.525076 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:37.524981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:37.525076 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:37.525041    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:37.537901 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:37.537878    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:38.211295 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:38.211261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" event={"ID":"22865335-cc68-4c31-87e1-de0d0e837f74","Type":"ContainerStarted","Data":"fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd"}
Sep 09 21:07:38.247692 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:38.247660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:07:39.138077 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:39.138035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:39.138296 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:39.138214    2578 patch_prober.go:28] interesting pod/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 21:07:39.138374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:39.138284    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 21:07:40.137818 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:40.137776    2578 patch_prober.go:28] interesting pod/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 21:07:40.138475 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:40.138443    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 21:07:41.137793 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:41.137754    2578 patch_prober.go:28] interesting pod/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 21:07:41.138088 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:41.138060    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 21:07:42.100640 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:42.100611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:07:42.137662 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:42.137625    2578 patch_prober.go:28] interesting pod/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 21:07:42.137921 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:42.137894    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 21:07:43.080547 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:43.080518    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:43.137074 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:43.137042    2578 patch_prober.go:28] interesting pod/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 21:07:43.137356 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:43.137332    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 21:07:44.141225 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:44.141195    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:44.142287 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:44.142261    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:07:52.907225 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:07:52.907194    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:08:08.815440 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:08.815351    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:08:20.906111 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:20.906065    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:08:21.568414 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.568379    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"]
Sep 09 21:08:21.568703 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.568650    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln" podUID="8450b7b3-85fd-4f11-bea7-7b906a6baad6" containerName="maas-api" containerID="cri-o://eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf" gracePeriod=30
Sep 09 21:08:21.808416 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.808388    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:08:21.859928 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.859891    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-maas-api-tls\") pod \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") "
Sep 09 21:08:21.860079 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.859970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhqrj\" (UniqueName: \"kubernetes.io/projected/8450b7b3-85fd-4f11-bea7-7b906a6baad6-kube-api-access-jhqrj\") pod \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") "
Sep 09 21:08:21.860079 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.860009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8450b7b3-85fd-4f11-bea7-7b906a6baad6-openshift-service-ca\") pod \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") "
Sep 09 21:08:21.860079 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.860062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-metrics-certs\") pod \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\" (UID: \"8450b7b3-85fd-4f11-bea7-7b906a6baad6\") "
Sep 09 21:08:21.860457 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.860426    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8450b7b3-85fd-4f11-bea7-7b906a6baad6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8450b7b3-85fd-4f11-bea7-7b906a6baad6" (UID: "8450b7b3-85fd-4f11-bea7-7b906a6baad6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:21.861938 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.861916    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8450b7b3-85fd-4f11-bea7-7b906a6baad6" (UID: "8450b7b3-85fd-4f11-bea7-7b906a6baad6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:21.862035 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.861971    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8450b7b3-85fd-4f11-bea7-7b906a6baad6" (UID: "8450b7b3-85fd-4f11-bea7-7b906a6baad6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:21.862035 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.862009    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8450b7b3-85fd-4f11-bea7-7b906a6baad6-kube-api-access-jhqrj" (OuterVolumeSpecName: "kube-api-access-jhqrj") pod "8450b7b3-85fd-4f11-bea7-7b906a6baad6" (UID: "8450b7b3-85fd-4f11-bea7-7b906a6baad6"). InnerVolumeSpecName "kube-api-access-jhqrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:21.961538 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.961509    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8450b7b3-85fd-4f11-bea7-7b906a6baad6-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:21.961538 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.961534    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:21.961538 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.961545    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8450b7b3-85fd-4f11-bea7-7b906a6baad6-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:21.962001 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:21.961553    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqrj\" (UniqueName: \"kubernetes.io/projected/8450b7b3-85fd-4f11-bea7-7b906a6baad6-kube-api-access-jhqrj\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:22.434227 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:22.434169    2578 generic.go:356] "Generic (PLEG): container finished" podID="8450b7b3-85fd-4f11-bea7-7b906a6baad6" containerID="eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf" exitCode=0
Sep 09 21:08:22.434227 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:22.434230    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"
Sep 09 21:08:22.434468 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:22.434253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln" event={"ID":"8450b7b3-85fd-4f11-bea7-7b906a6baad6","Type":"ContainerDied","Data":"eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf"}
Sep 09 21:08:22.434468 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:22.434306    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln" event={"ID":"8450b7b3-85fd-4f11-bea7-7b906a6baad6","Type":"ContainerDied","Data":"aeba6c0c661e322dafca391d3d6f55eb6483e0b2991c95ada2addeebc9fa6b2b"}
Sep 09 21:08:22.435128 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:22.434335    2578 scope.go:117] "RemoveContainer" containerID="eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf"
Sep 09 21:08:22.448065 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:22.448046    2578 scope.go:117] "RemoveContainer" containerID="eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf"
Sep 09 21:08:22.448358 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:08:22.448340    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf\": container with ID starting with eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf not found: ID does not exist" containerID="eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf"
Sep 09 21:08:22.448424 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:22.448367    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf"} err="failed to get container status \"eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf\": rpc error: code = NotFound desc = could not find container \"eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf\": container with ID starting with eb07ce729f68dd58e55e4702dfb0ff9d13e9cbfc273d06d364a806c38d1a3adf not found: ID does not exist"
Sep 09 21:08:22.460390 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:22.460369    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"]
Sep 09 21:08:22.466692 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:22.466673    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0066f8-865ddc554f-gppln"]
Sep 09 21:08:23.842972 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:23.842938    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8450b7b3-85fd-4f11-bea7-7b906a6baad6" path="/var/lib/kubelet/pods/8450b7b3-85fd-4f11-bea7-7b906a6baad6/volumes"
Sep 09 21:08:29.153005 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:29.152964    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"]
Sep 09 21:08:29.153452 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:29.153277    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy" containerID="cri-o://fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd" gracePeriod=30
Sep 09 21:08:29.921675 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:29.921636    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"]
Sep 09 21:08:29.922005 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:29.921988    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8450b7b3-85fd-4f11-bea7-7b906a6baad6" containerName="maas-api"
Sep 09 21:08:29.922331 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:29.922008    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8450b7b3-85fd-4f11-bea7-7b906a6baad6" containerName="maas-api"
Sep 09 21:08:29.922331 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:29.922110    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8450b7b3-85fd-4f11-bea7-7b906a6baad6" containerName="maas-api"
Sep 09 21:08:29.926535 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:29.926514    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:29.929475 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:29.929459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-46ba3a-openshift-default-dockercfg-jsp4c\""
Sep 09 21:08:29.938429 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:29.938408    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"]
Sep 09 21:08:30.031363 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.031322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-data\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.031587 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.031399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-certs\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.031587 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.031429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-envoy\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.031587 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.031457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istiod-ca-cert\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.031587 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.031542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-socket\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.031587 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.031579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbh6t\" (UniqueName: \"kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-kube-api-access-tbh6t\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.031880 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.031608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-credential-socket\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.031880 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.031676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-podinfo\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.031880 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.031700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-token\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.132266 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.132237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-certs\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.132266 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.132269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-envoy\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.132503 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.132289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istiod-ca-cert\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.132503 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.132314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-socket\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.132503 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.132329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbh6t\" (UniqueName: \"kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-kube-api-access-tbh6t\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.132503 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.132350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-credential-socket\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.132503 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.132395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-podinfo\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.132752 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.132520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-token\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.132752 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.132597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-data\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.132752 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.132692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-certs\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.132904 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.132845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-socket\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.133059 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.133040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-data\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.133059 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.133059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-credential-socket\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.133214 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.133065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istiod-ca-cert\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.134659 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.134639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-envoy\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.134843 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.134826    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-podinfo\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.139437 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.139417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-token\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.139713 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.139695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbh6t\" (UniqueName: \"kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-kube-api-access-tbh6t\") pod \"e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") " pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.236182 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.236076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:30.512927 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.512846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"]
Sep 09 21:08:30.515683 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:08:30.515653    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2af1e74_384d_4d2e_b1a5_8f5971a4b660.slice/crio-be7777662bc91aac6b193013d7d4e22901884d3ce112958518e14bb40ee13e74 WatchSource:0}: Error finding container be7777662bc91aac6b193013d7d4e22901884d3ce112958518e14bb40ee13e74: Status 404 returned error can't find the container with id be7777662bc91aac6b193013d7d4e22901884d3ce112958518e14bb40ee13e74
Sep 09 21:08:30.517752 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.517713    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:08:30.517823 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.517798    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:08:30.517859 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:30.517839    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:08:31.478116 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:31.478083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj" event={"ID":"f2af1e74-384d-4d2e-b1a5-8f5971a4b660","Type":"ContainerStarted","Data":"e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9"}
Sep 09 21:08:31.478501 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:31.478147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj" event={"ID":"f2af1e74-384d-4d2e-b1a5-8f5971a4b660","Type":"ContainerStarted","Data":"be7777662bc91aac6b193013d7d4e22901884d3ce112958518e14bb40ee13e74"}
Sep 09 21:08:31.500623 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:31.500571    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj" podStartSLOduration=2.500557004 podStartE2EDuration="2.500557004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:31.498696183 +0000 UTC m=+1188.153476345" watchObservedRunningTime="2026-09-09 21:08:31.500557004 +0000 UTC m=+1188.155337162"
Sep 09 21:08:32.236781 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:32.236744    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:32.241638 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:32.241613    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:32.482950 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:32.482911    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:32.483908 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:32.483889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:08:33.007803 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:33.007772    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:08:34.412493 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.412464    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:08:34.471970 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.471889    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-credential-socket\") pod \"22865335-cc68-4c31-87e1-de0d0e837f74\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") "
Sep 09 21:08:34.471970 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.471946    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45ffn\" (UniqueName: \"kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-kube-api-access-45ffn\") pod \"22865335-cc68-4c31-87e1-de0d0e837f74\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") "
Sep 09 21:08:34.472222 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22865335-cc68-4c31-87e1-de0d0e837f74-istio-podinfo\") pod \"22865335-cc68-4c31-87e1-de0d0e837f74\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") "
Sep 09 21:08:34.472222 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-socket\") pod \"22865335-cc68-4c31-87e1-de0d0e837f74\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") "
Sep 09 21:08:34.472222 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472103    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-istio-token\") pod \"22865335-cc68-4c31-87e1-de0d0e837f74\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") "
Sep 09 21:08:34.472222 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472127    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-data\") pod \"22865335-cc68-4c31-87e1-de0d0e837f74\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") "
Sep 09 21:08:34.472222 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472163    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-certs\") pod \"22865335-cc68-4c31-87e1-de0d0e837f74\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") "
Sep 09 21:08:34.472222 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472198    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22865335-cc68-4c31-87e1-de0d0e837f74" (UID: "22865335-cc68-4c31-87e1-de0d0e837f74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:34.472640 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-envoy\") pod \"22865335-cc68-4c31-87e1-de0d0e837f74\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") "
Sep 09 21:08:34.472640 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22865335-cc68-4c31-87e1-de0d0e837f74-istiod-ca-cert\") pod \"22865335-cc68-4c31-87e1-de0d0e837f74\" (UID: \"22865335-cc68-4c31-87e1-de0d0e837f74\") "
Sep 09 21:08:34.472640 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22865335-cc68-4c31-87e1-de0d0e837f74" (UID: "22865335-cc68-4c31-87e1-de0d0e837f74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:34.472640 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472440    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22865335-cc68-4c31-87e1-de0d0e837f74" (UID: "22865335-cc68-4c31-87e1-de0d0e837f74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:34.472640 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-data" (OuterVolumeSpecName: "istio-data") pod "22865335-cc68-4c31-87e1-de0d0e837f74" (UID: "22865335-cc68-4c31-87e1-de0d0e837f74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:34.472896 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472735    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:34.472896 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472755    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:34.472896 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472757    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22865335-cc68-4c31-87e1-de0d0e837f74-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22865335-cc68-4c31-87e1-de0d0e837f74" (UID: "22865335-cc68-4c31-87e1-de0d0e837f74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:34.472896 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472769    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:34.472896 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.472781    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:34.474421 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.474390    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-kube-api-access-45ffn" (OuterVolumeSpecName: "kube-api-access-45ffn") pod "22865335-cc68-4c31-87e1-de0d0e837f74" (UID: "22865335-cc68-4c31-87e1-de0d0e837f74"). InnerVolumeSpecName "kube-api-access-45ffn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:34.474421 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.474397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-istio-token" (OuterVolumeSpecName: "istio-token") pod "22865335-cc68-4c31-87e1-de0d0e837f74" (UID: "22865335-cc68-4c31-87e1-de0d0e837f74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:34.474595 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.474448    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22865335-cc68-4c31-87e1-de0d0e837f74-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22865335-cc68-4c31-87e1-de0d0e837f74" (UID: "22865335-cc68-4c31-87e1-de0d0e837f74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:08:34.474595 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.474529    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22865335-cc68-4c31-87e1-de0d0e837f74" (UID: "22865335-cc68-4c31-87e1-de0d0e837f74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:34.494126 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.494098    2578 generic.go:356] "Generic (PLEG): container finished" podID="22865335-cc68-4c31-87e1-de0d0e837f74" containerID="fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd" exitCode=0
Sep 09 21:08:34.494248 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.494145    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"
Sep 09 21:08:34.494248 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.494204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" event={"ID":"22865335-cc68-4c31-87e1-de0d0e837f74","Type":"ContainerDied","Data":"fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd"}
Sep 09 21:08:34.494326 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.494260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k" event={"ID":"22865335-cc68-4c31-87e1-de0d0e837f74","Type":"ContainerDied","Data":"12e6a2cb5e5a8dc9576fdddddb8d4aa2e9de65ad73b6273c21637e35351ad361"}
Sep 09 21:08:34.494326 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.494289    2578 scope.go:117] "RemoveContainer" containerID="fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd"
Sep 09 21:08:34.502743 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.502727    2578 scope.go:117] "RemoveContainer" containerID="07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913"
Sep 09 21:08:34.510010 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.509994    2578 scope.go:117] "RemoveContainer" containerID="fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd"
Sep 09 21:08:34.510285 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:08:34.510264    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd\": container with ID starting with fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd not found: ID does not exist" containerID="fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd"
Sep 09 21:08:34.510362 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.510299    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd"} err="failed to get container status \"fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd\": rpc error: code = NotFound desc = could not find container \"fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd\": container with ID starting with fdd4e2efd5af5828baa67148c33981e2b01d80a952083975864edbbf91250fdd not found: ID does not exist"
Sep 09 21:08:34.510362 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.510325    2578 scope.go:117] "RemoveContainer" containerID="07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913"
Sep 09 21:08:34.510598 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:08:34.510581    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913\": container with ID starting with 07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913 not found: ID does not exist" containerID="07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913"
Sep 09 21:08:34.510648 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.510603    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913"} err="failed to get container status \"07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913\": rpc error: code = NotFound desc = could not find container \"07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913\": container with ID starting with 07782c450eee1c42e081ef7fa88b7bedd010f24cb63f1b1efb26f3755ead7913 not found: ID does not exist"
Sep 09 21:08:34.516491 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.516471    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"]
Sep 09 21:08:34.520681 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.520662    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-0066f8-openshift-default-6958f44c84-wxz9k"]
Sep 09 21:08:34.573712 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.573679    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:34.573712 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.573711    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22865335-cc68-4c31-87e1-de0d0e837f74-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:34.573899 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.573725    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22865335-cc68-4c31-87e1-de0d0e837f74-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:34.573899 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.573738    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45ffn\" (UniqueName: \"kubernetes.io/projected/22865335-cc68-4c31-87e1-de0d0e837f74-kube-api-access-45ffn\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:34.573899 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:34.573751    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22865335-cc68-4c31-87e1-de0d0e837f74-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:08:35.843763 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:35.843731    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" path="/var/lib/kubelet/pods/22865335-cc68-4c31-87e1-de0d0e837f74/volumes"
Sep 09 21:08:50.116237 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:50.116203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:08:54.002979 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:54.002940    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"]
Sep 09 21:08:55.330281 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:55.330240    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"]
Sep 09 21:08:55.330726 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:55.330456    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj" podUID="f2af1e74-384d-4d2e-b1a5-8f5971a4b660" containerName="istio-proxy" containerID="cri-o://e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9" gracePeriod=30
Sep 09 21:08:56.493905 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.493876    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"]
Sep 09 21:08:56.676324 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.676293    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"]
Sep 09 21:08:56.676591 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.676579    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy"
Sep 09 21:08:56.676654 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.676592    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy"
Sep 09 21:08:56.676693 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.676654    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy"
Sep 09 21:08:56.676693 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.676662    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy"
Sep 09 21:08:56.680852 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.680834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"
Sep 09 21:08:56.683249 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.683224    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:08:56.683249 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.683231    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6l2wq\""
Sep 09 21:08:56.689625 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.689598    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"]
Sep 09 21:08:56.870544 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.870513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7272bc3b-94ba-4cb9-ae14-86407ba63c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9\" (UID: \"7272bc3b-94ba-4cb9-ae14-86407ba63c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"
Sep 09 21:08:56.970975 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.970939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7272bc3b-94ba-4cb9-ae14-86407ba63c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9\" (UID: \"7272bc3b-94ba-4cb9-ae14-86407ba63c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"
Sep 09 21:08:56.971593 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.971574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7272bc3b-94ba-4cb9-ae14-86407ba63c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9\" (UID: \"7272bc3b-94ba-4cb9-ae14-86407ba63c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"
Sep 09 21:08:56.991955 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:56.991935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"
Sep 09 21:08:57.259198 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:57.259064    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"]
Sep 09 21:08:57.261982 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:08:57.261956    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7272bc3b_94ba_4cb9_ae14_86407ba63c72.slice/crio-95a0fd3814bc656207e831706cef5b312da0e3e4c250856d4ad23d72bddfe013 WatchSource:0}: Error finding container 95a0fd3814bc656207e831706cef5b312da0e3e4c250856d4ad23d72bddfe013: Status 404 returned error can't find the container with id 95a0fd3814bc656207e831706cef5b312da0e3e4c250856d4ad23d72bddfe013
Sep 09 21:08:57.599000 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:57.598967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9" event={"ID":"7272bc3b-94ba-4cb9-ae14-86407ba63c72","Type":"ContainerStarted","Data":"8007458adef6e7ee36a2b46921a98d076fab60bd4d3b27c1673ec1eebea6a4c8"}
Sep 09 21:08:57.599379 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:57.599019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9" event={"ID":"7272bc3b-94ba-4cb9-ae14-86407ba63c72","Type":"ContainerStarted","Data":"95a0fd3814bc656207e831706cef5b312da0e3e4c250856d4ad23d72bddfe013"}
Sep 09 21:08:57.613892 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:57.613852    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9" podStartSLOduration=1.6138369369999999 podStartE2EDuration="1.613836937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:57.613795135 +0000 UTC m=+1214.268575292" watchObservedRunningTime="2026-09-09 21:08:57.613836937 +0000 UTC m=+1214.268617095"
Sep 09 21:08:58.605351 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:58.605318    2578 generic.go:356] "Generic (PLEG): container finished" podID="7272bc3b-94ba-4cb9-ae14-86407ba63c72" containerID="8007458adef6e7ee36a2b46921a98d076fab60bd4d3b27c1673ec1eebea6a4c8" exitCode=0
Sep 09 21:08:58.605720 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:58.605381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9" event={"ID":"7272bc3b-94ba-4cb9-ae14-86407ba63c72","Type":"ContainerDied","Data":"8007458adef6e7ee36a2b46921a98d076fab60bd4d3b27c1673ec1eebea6a4c8"}
Sep 09 21:08:59.734920 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:59.734895    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"
Sep 09 21:08:59.790298 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:59.790264    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7272bc3b-94ba-4cb9-ae14-86407ba63c72-maas-api-service-ca\") pod \"7272bc3b-94ba-4cb9-ae14-86407ba63c72\" (UID: \"7272bc3b-94ba-4cb9-ae14-86407ba63c72\") "
Sep 09 21:08:59.790601 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:59.790579    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7272bc3b-94ba-4cb9-ae14-86407ba63c72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7272bc3b-94ba-4cb9-ae14-86407ba63c72" (UID: "7272bc3b-94ba-4cb9-ae14-86407ba63c72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:59.891445 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:08:59.891374    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7272bc3b-94ba-4cb9-ae14-86407ba63c72-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:00.583003 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.582981    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:09:00.597834 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.597787    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-credential-socket\") pod \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") "
Sep 09 21:09:00.597834 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.597818    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-data\") pod \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") "
Sep 09 21:09:00.597956 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.597852    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-envoy\") pod \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") "
Sep 09 21:09:00.597956 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.597891    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbh6t\" (UniqueName: \"kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-kube-api-access-tbh6t\") pod \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") "
Sep 09 21:09:00.597956 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.597919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-podinfo\") pod \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") "
Sep 09 21:09:00.598071 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.597968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istiod-ca-cert\") pod \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") "
Sep 09 21:09:00.598071 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.597996    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-token\") pod \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") "
Sep 09 21:09:00.598071 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.598022    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-socket\") pod \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") "
Sep 09 21:09:00.598071 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.598062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-certs\") pod \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\" (UID: \"f2af1e74-384d-4d2e-b1a5-8f5971a4b660\") "
Sep 09 21:09:00.598071 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.598062    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2af1e74-384d-4d2e-b1a5-8f5971a4b660" (UID: "f2af1e74-384d-4d2e-b1a5-8f5971a4b660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:00.598310 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.598079    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2af1e74-384d-4d2e-b1a5-8f5971a4b660" (UID: "f2af1e74-384d-4d2e-b1a5-8f5971a4b660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:00.598372 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.598320    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2af1e74-384d-4d2e-b1a5-8f5971a4b660" (UID: "f2af1e74-384d-4d2e-b1a5-8f5971a4b660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:00.598372 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.598335    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:00.598372 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.598354    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:00.598528 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.598381    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2af1e74-384d-4d2e-b1a5-8f5971a4b660" (UID: "f2af1e74-384d-4d2e-b1a5-8f5971a4b660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:00.598581 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.598552    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2af1e74-384d-4d2e-b1a5-8f5971a4b660" (UID: "f2af1e74-384d-4d2e-b1a5-8f5971a4b660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:00.599952 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.599931    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2af1e74-384d-4d2e-b1a5-8f5971a4b660" (UID: "f2af1e74-384d-4d2e-b1a5-8f5971a4b660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:00.600160 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.600142    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2af1e74-384d-4d2e-b1a5-8f5971a4b660" (UID: "f2af1e74-384d-4d2e-b1a5-8f5971a4b660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:00.600383 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.600361    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2af1e74-384d-4d2e-b1a5-8f5971a4b660" (UID: "f2af1e74-384d-4d2e-b1a5-8f5971a4b660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:09:00.600450 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.600404    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-kube-api-access-tbh6t" (OuterVolumeSpecName: "kube-api-access-tbh6t") pod "f2af1e74-384d-4d2e-b1a5-8f5971a4b660" (UID: "f2af1e74-384d-4d2e-b1a5-8f5971a4b660"). InnerVolumeSpecName "kube-api-access-tbh6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:00.614863 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.614836    2578 generic.go:356] "Generic (PLEG): container finished" podID="f2af1e74-384d-4d2e-b1a5-8f5971a4b660" containerID="e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9" exitCode=0
Sep 09 21:09:00.614961 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.614881    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"
Sep 09 21:09:00.614961 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.614917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj" event={"ID":"f2af1e74-384d-4d2e-b1a5-8f5971a4b660","Type":"ContainerDied","Data":"e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9"}
Sep 09 21:09:00.615047 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.614981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj" event={"ID":"f2af1e74-384d-4d2e-b1a5-8f5971a4b660","Type":"ContainerDied","Data":"be7777662bc91aac6b193013d7d4e22901884d3ce112958518e14bb40ee13e74"}
Sep 09 21:09:00.615047 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.615016    2578 scope.go:117] "RemoveContainer" containerID="e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9"
Sep 09 21:09:00.616475 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.616457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9" event={"ID":"7272bc3b-94ba-4cb9-ae14-86407ba63c72","Type":"ContainerDied","Data":"95a0fd3814bc656207e831706cef5b312da0e3e4c250856d4ad23d72bddfe013"}
Sep 09 21:09:00.616555 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.616481    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"
Sep 09 21:09:00.616555 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.616498    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95a0fd3814bc656207e831706cef5b312da0e3e4c250856d4ad23d72bddfe013"
Sep 09 21:09:00.625924 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.625906    2578 scope.go:117] "RemoveContainer" containerID="e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9"
Sep 09 21:09:00.626221 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:09:00.626154    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9\": container with ID starting with e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9 not found: ID does not exist" containerID="e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9"
Sep 09 21:09:00.626221 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.626210    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9"} err="failed to get container status \"e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9\": rpc error: code = NotFound desc = could not find container \"e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9\": container with ID starting with e0f4e75e7e72de0074374b3577aacf47b6b4c76b6237ddbe19b3e0c2539305c9 not found: ID does not exist"
Sep 09 21:09:00.640367 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.640342    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"]
Sep 09 21:09:00.652244 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.652222    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-46ba3a-openshift-default-578bbf8498-25lsj"]
Sep 09 21:09:00.698816 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.698794    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:00.698816 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.698814    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:00.698946 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.698823    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:00.698946 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.698832    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:00.698946 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.698841    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:00.698946 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.698850    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbh6t\" (UniqueName: \"kubernetes.io/projected/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-kube-api-access-tbh6t\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:00.698946 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:00.698858    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2af1e74-384d-4d2e-b1a5-8f5971a4b660-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:01.252656 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.252621    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"]
Sep 09 21:09:01.253106 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.253075    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2af1e74-384d-4d2e-b1a5-8f5971a4b660" containerName="istio-proxy"
Sep 09 21:09:01.253106 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.253096    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2af1e74-384d-4d2e-b1a5-8f5971a4b660" containerName="istio-proxy"
Sep 09 21:09:01.253258 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.253129    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy"
Sep 09 21:09:01.253258 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.253138    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22865335-cc68-4c31-87e1-de0d0e837f74" containerName="istio-proxy"
Sep 09 21:09:01.253258 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.253150    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7272bc3b-94ba-4cb9-ae14-86407ba63c72" containerName="revoke-keys"
Sep 09 21:09:01.253258 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.253160    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7272bc3b-94ba-4cb9-ae14-86407ba63c72" containerName="revoke-keys"
Sep 09 21:09:01.253479 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.253266    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7272bc3b-94ba-4cb9-ae14-86407ba63c72" containerName="revoke-keys"
Sep 09 21:09:01.253479 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.253282    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2af1e74-384d-4d2e-b1a5-8f5971a4b660" containerName="istio-proxy"
Sep 09 21:09:01.257584 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.257561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.260264 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.260243    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-57783aa9-openshift-default-dockercfg-xr89l\""
Sep 09 21:09:01.270948 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.270925    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"]
Sep 09 21:09:01.304156 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.304122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-certs\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.304156 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.304156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhz8t\" (UniqueName: \"kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-kube-api-access-nhz8t\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.304351 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.304196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-socket\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.304351 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.304221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee0819d-2938-41fd-abb7-1f456fc29753-istiod-ca-cert\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.304351 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.304243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-envoy\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.304351 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.304275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-podinfo\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.304351 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.304323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-token\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.304351 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.304344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-data\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.304539 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.304437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-credential-socket\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405252 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-podinfo\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405428 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-token\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405428 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-data\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405428 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-credential-socket\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405428 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-certs\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405623 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhz8t\" (UniqueName: \"kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-kube-api-access-nhz8t\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405623 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-socket\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405623 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee0819d-2938-41fd-abb7-1f456fc29753-istiod-ca-cert\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405796 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-envoy\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405796 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-data\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405883 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-socket\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405939 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-credential-socket\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.405996 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.405967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-certs\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.406342 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.406325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee0819d-2938-41fd-abb7-1f456fc29753-istiod-ca-cert\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.407693 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.407673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-envoy\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.407783 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.407732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-podinfo\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.413403 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.413383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-token\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.413522 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.413447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhz8t\" (UniqueName: \"kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-kube-api-access-nhz8t\") pod \"e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") " pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.566939 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.566853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:01.841739 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.841165    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2af1e74-384d-4d2e-b1a5-8f5971a4b660" path="/var/lib/kubelet/pods/f2af1e74-384d-4d2e-b1a5-8f5971a4b660/volumes"
Sep 09 21:09:01.843475 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.843411    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"]
Sep 09 21:09:01.845818 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:09:01.845793    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee0819d_2938_41fd_abb7_1f456fc29753.slice/crio-e33c1ec89c607a5aae8bf3d00d163df68486077ba394aa0529075c9a7ccf6322 WatchSource:0}: Error finding container e33c1ec89c607a5aae8bf3d00d163df68486077ba394aa0529075c9a7ccf6322: Status 404 returned error can't find the container with id e33c1ec89c607a5aae8bf3d00d163df68486077ba394aa0529075c9a7ccf6322
Sep 09 21:09:01.847949 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.847920    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:09:01.848006 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.847979    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:09:01.848049 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:01.848009    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:09:02.631131 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:02.631091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" event={"ID":"1ee0819d-2938-41fd-abb7-1f456fc29753","Type":"ContainerStarted","Data":"c6445131e6d9e19a052c8afc61b041ee8e2ba72261eac6d6992b42dd03729dd5"}
Sep 09 21:09:02.631633 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:02.631155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" event={"ID":"1ee0819d-2938-41fd-abb7-1f456fc29753","Type":"ContainerStarted","Data":"e33c1ec89c607a5aae8bf3d00d163df68486077ba394aa0529075c9a7ccf6322"}
Sep 09 21:09:02.654374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:02.654321    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" podStartSLOduration=1.654308041 podStartE2EDuration="1.654308041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:02.652791092 +0000 UTC m=+1219.307571252" watchObservedRunningTime="2026-09-09 21:09:02.654308041 +0000 UTC m=+1219.309088198"
Sep 09 21:09:03.567595 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:03.567558    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:03.572461 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:03.572431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:03.635636 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:03.635603    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:03.636447 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:03.636428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:05.213921 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.213885    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:09:05.796024 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.795989    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"]
Sep 09 21:09:05.799784 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.799762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:05.802435 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.802416    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-57783aa9-serving-cert\""
Sep 09 21:09:05.802552 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.802468    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:09:05.803200 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.803165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gj4rp\""
Sep 09 21:09:05.803286 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.803227    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 21:09:05.803472 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.803456    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 21:09:05.803515 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.803469    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-gateway-identity\""
Sep 09 21:09:05.812068 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.812039    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"]
Sep 09 21:09:05.843455 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.843429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlhm5\" (UniqueName: \"kubernetes.io/projected/ce33ff71-1173-498a-9ac7-2fba603bda5b-kube-api-access-tlhm5\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:05.843600 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.843535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce33ff71-1173-498a-9ac7-2fba603bda5b-openshift-service-ca\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:05.843600 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.843578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-maas-api-tls\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:05.843683 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.843600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-metrics-certs\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:05.944515 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.944481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlhm5\" (UniqueName: \"kubernetes.io/projected/ce33ff71-1173-498a-9ac7-2fba603bda5b-kube-api-access-tlhm5\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:05.944723 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.944556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce33ff71-1173-498a-9ac7-2fba603bda5b-openshift-service-ca\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:05.944723 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.944586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-maas-api-tls\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:05.944723 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.944603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-metrics-certs\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:05.944723 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:09:05.944684    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-serving-cert: secret "maas-api-e2e-mt-57783aa9-serving-cert" not found
Sep 09 21:09:05.944937 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:09:05.944731    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-maas-api-tls podName:ce33ff71-1173-498a-9ac7-2fba603bda5b nodeName:}" failed. No retries permitted until 2026-09-09 21:09:06.444715729 +0000 UTC m=+1223.099495864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-maas-api-tls") pod "maas-api-e2e-mt-57783aa9-556648d477-kptdt" (UID: "ce33ff71-1173-498a-9ac7-2fba603bda5b") : secret "maas-api-e2e-mt-57783aa9-serving-cert" not found
Sep 09 21:09:05.945333 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.945303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce33ff71-1173-498a-9ac7-2fba603bda5b-openshift-service-ca\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:05.946996 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.946977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-metrics-certs\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:05.953426 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:05.953405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlhm5\" (UniqueName: \"kubernetes.io/projected/ce33ff71-1173-498a-9ac7-2fba603bda5b-kube-api-access-tlhm5\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:06.447673 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:06.447639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-maas-api-tls\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:06.449920 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:06.449902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-maas-api-tls\") pod \"maas-api-e2e-mt-57783aa9-556648d477-kptdt\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:06.710851 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:06.710773    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:06.981603 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:06.981577    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"]
Sep 09 21:09:07.293984 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:07.293899    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"]
Sep 09 21:09:07.295976 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:07.295949    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9a54ea-xkpf9"]
Sep 09 21:09:07.654372 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:07.654339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt" event={"ID":"ce33ff71-1173-498a-9ac7-2fba603bda5b","Type":"ContainerStarted","Data":"105a4e8d89d1cf4804ee9d2c66b24caee073cbdc1e7e1386f998eda253a818c3"}
Sep 09 21:09:07.841012 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:07.840983    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7272bc3b-94ba-4cb9-ae14-86407ba63c72" path="/var/lib/kubelet/pods/7272bc3b-94ba-4cb9-ae14-86407ba63c72/volumes"
Sep 09 21:09:08.660116 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:08.660084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt" event={"ID":"ce33ff71-1173-498a-9ac7-2fba603bda5b","Type":"ContainerStarted","Data":"e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6"}
Sep 09 21:09:08.660533 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:08.660162    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:08.680253 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:08.680205    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt" podStartSLOduration=3.005487547 podStartE2EDuration="3.680189458s" totalImagesPullingTime="674.701911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:08.678968762 +0000 UTC m=+1225.333748929" watchObservedRunningTime="2026-09-09 21:09:08.680189458 +0000 UTC m=+1225.334969608"
Sep 09 21:09:09.021987 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.021905    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5" podUID="df08a066-11dc-43df-8245-c9906f780c93" containerName="main" containerID="cri-o://495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f" gracePeriod=45
Sep 09 21:09:09.256275 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.256252    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:09:09.373894 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.373867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-dshm\") pod \"df08a066-11dc-43df-8245-c9906f780c93\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") "
Sep 09 21:09:09.374084 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.373913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-home\") pod \"df08a066-11dc-43df-8245-c9906f780c93\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") "
Sep 09 21:09:09.374084 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.373934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df08a066-11dc-43df-8245-c9906f780c93-tls-certs\") pod \"df08a066-11dc-43df-8245-c9906f780c93\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") "
Sep 09 21:09:09.374084 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.373950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-model-cache\") pod \"df08a066-11dc-43df-8245-c9906f780c93\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") "
Sep 09 21:09:09.374084 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.374005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2584f\" (UniqueName: \"kubernetes.io/projected/df08a066-11dc-43df-8245-c9906f780c93-kube-api-access-2584f\") pod \"df08a066-11dc-43df-8245-c9906f780c93\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") "
Sep 09 21:09:09.374084 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.374042    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-tmp-dir\") pod \"df08a066-11dc-43df-8245-c9906f780c93\" (UID: \"df08a066-11dc-43df-8245-c9906f780c93\") "
Sep 09 21:09:09.374383 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.374236    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-home" (OuterVolumeSpecName: "home") pod "df08a066-11dc-43df-8245-c9906f780c93" (UID: "df08a066-11dc-43df-8245-c9906f780c93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:09.374383 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.374309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-model-cache" (OuterVolumeSpecName: "model-cache") pod "df08a066-11dc-43df-8245-c9906f780c93" (UID: "df08a066-11dc-43df-8245-c9906f780c93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:09.374383 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.374340    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-home\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:09.374517 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.374380    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df08a066-11dc-43df-8245-c9906f780c93" (UID: "df08a066-11dc-43df-8245-c9906f780c93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:09.376284 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.376259    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-dshm" (OuterVolumeSpecName: "dshm") pod "df08a066-11dc-43df-8245-c9906f780c93" (UID: "df08a066-11dc-43df-8245-c9906f780c93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:09.376394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.376294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df08a066-11dc-43df-8245-c9906f780c93-kube-api-access-2584f" (OuterVolumeSpecName: "kube-api-access-2584f") pod "df08a066-11dc-43df-8245-c9906f780c93" (UID: "df08a066-11dc-43df-8245-c9906f780c93"). InnerVolumeSpecName "kube-api-access-2584f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:09.376394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.376366    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df08a066-11dc-43df-8245-c9906f780c93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df08a066-11dc-43df-8245-c9906f780c93" (UID: "df08a066-11dc-43df-8245-c9906f780c93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:09.474998 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.474976    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2584f\" (UniqueName: \"kubernetes.io/projected/df08a066-11dc-43df-8245-c9906f780c93-kube-api-access-2584f\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:09.475068 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.475000    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-tmp-dir\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:09.475068 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.475010    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-dshm\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:09.475068 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.475018    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df08a066-11dc-43df-8245-c9906f780c93-tls-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:09.475068 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.475026    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df08a066-11dc-43df-8245-c9906f780c93-model-cache\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:09.665893 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.665806    2578 generic.go:356] "Generic (PLEG): container finished" podID="df08a066-11dc-43df-8245-c9906f780c93" containerID="495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f" exitCode=0
Sep 09 21:09:09.665893 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.665856    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"
Sep 09 21:09:09.666425 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.665886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5" event={"ID":"df08a066-11dc-43df-8245-c9906f780c93","Type":"ContainerDied","Data":"495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f"}
Sep 09 21:09:09.666425 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.665928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5" event={"ID":"df08a066-11dc-43df-8245-c9906f780c93","Type":"ContainerDied","Data":"8bfae3fc78e32c9c41c89ca55182028c9069574e44ceda7789e879c9bd29af99"}
Sep 09 21:09:09.666425 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.665946    2578 scope.go:117] "RemoveContainer" containerID="495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f"
Sep 09 21:09:09.674485 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.674448    2578 scope.go:117] "RemoveContainer" containerID="495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f"
Sep 09 21:09:09.674757 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:09:09.674740    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f\": container with ID starting with 495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f not found: ID does not exist" containerID="495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f"
Sep 09 21:09:09.674803 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.674766    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f"} err="failed to get container status \"495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f\": rpc error: code = NotFound desc = could not find container \"495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f\": container with ID starting with 495804a6032ea76edc70492b16702401e2a5c44c9f424fc73fcb11c729f40b6f not found: ID does not exist"
Sep 09 21:09:09.696110 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.696083    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"]
Sep 09 21:09:09.707143 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.707125    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9a54ea/test-model-9a54ea-kserve-797f76d4f9-s58b5"]
Sep 09 21:09:09.846305 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:09.846271    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df08a066-11dc-43df-8245-c9906f780c93" path="/var/lib/kubelet/pods/df08a066-11dc-43df-8245-c9906f780c93/volumes"
Sep 09 21:09:11.512763 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.512717    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29" podUID="20a51d8c-e4d7-4024-95ea-61d42d8ddebe" containerName="main" containerID="cri-o://0866524d079243d578fafd0b740636c3d2e8b93e4d972af6195be696a5ea133e" gracePeriod=45
Sep 09 21:09:11.679598 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.679344    2578 generic.go:356] "Generic (PLEG): container finished" podID="20a51d8c-e4d7-4024-95ea-61d42d8ddebe" containerID="0866524d079243d578fafd0b740636c3d2e8b93e4d972af6195be696a5ea133e" exitCode=0
Sep 09 21:09:11.679598 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.679423    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29" event={"ID":"20a51d8c-e4d7-4024-95ea-61d42d8ddebe","Type":"ContainerDied","Data":"0866524d079243d578fafd0b740636c3d2e8b93e4d972af6195be696a5ea133e"}
Sep 09 21:09:11.779340 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.779316    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:09:11.896590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.896560    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-model-cache\") pod \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") "
Sep 09 21:09:11.896764 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.896595    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-home\") pod \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") "
Sep 09 21:09:11.896764 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.896632    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4dp9\" (UniqueName: \"kubernetes.io/projected/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-kube-api-access-g4dp9\") pod \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") "
Sep 09 21:09:11.896764 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.896675    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tmp-dir\") pod \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") "
Sep 09 21:09:11.896764 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.896712    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tls-certs\") pod \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") "
Sep 09 21:09:11.896764 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.896743    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-dshm\") pod \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\" (UID: \"20a51d8c-e4d7-4024-95ea-61d42d8ddebe\") "
Sep 09 21:09:11.897049 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.896834    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-home" (OuterVolumeSpecName: "home") pod "20a51d8c-e4d7-4024-95ea-61d42d8ddebe" (UID: "20a51d8c-e4d7-4024-95ea-61d42d8ddebe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:11.897049 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.896861    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-model-cache" (OuterVolumeSpecName: "model-cache") pod "20a51d8c-e4d7-4024-95ea-61d42d8ddebe" (UID: "20a51d8c-e4d7-4024-95ea-61d42d8ddebe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:11.897049 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.896949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20a51d8c-e4d7-4024-95ea-61d42d8ddebe" (UID: "20a51d8c-e4d7-4024-95ea-61d42d8ddebe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:11.897203 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.897049    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-model-cache\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:11.897203 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.897064    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-home\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:11.897203 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.897079    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tmp-dir\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:11.898997 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.898966    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-kube-api-access-g4dp9" (OuterVolumeSpecName: "kube-api-access-g4dp9") pod "20a51d8c-e4d7-4024-95ea-61d42d8ddebe" (UID: "20a51d8c-e4d7-4024-95ea-61d42d8ddebe"). InnerVolumeSpecName "kube-api-access-g4dp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:11.898997 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.898984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20a51d8c-e4d7-4024-95ea-61d42d8ddebe" (UID: "20a51d8c-e4d7-4024-95ea-61d42d8ddebe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:11.899159 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.899048    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-dshm" (OuterVolumeSpecName: "dshm") pod "20a51d8c-e4d7-4024-95ea-61d42d8ddebe" (UID: "20a51d8c-e4d7-4024-95ea-61d42d8ddebe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:11.997659 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.997604    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-tls-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:11.997659 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.997651    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-dshm\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:11.997659 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:11.997665    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4dp9\" (UniqueName: \"kubernetes.io/projected/20a51d8c-e4d7-4024-95ea-61d42d8ddebe-kube-api-access-g4dp9\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:12.685708 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:12.685680    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"
Sep 09 21:09:12.686096 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:12.685683    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29" event={"ID":"20a51d8c-e4d7-4024-95ea-61d42d8ddebe","Type":"ContainerDied","Data":"dd4c6622cbb6af6aa6188c8749fbdbdd90c82097e8372927c611b6db9674b739"}
Sep 09 21:09:12.686096 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:12.685802    2578 scope.go:117] "RemoveContainer" containerID="0866524d079243d578fafd0b740636c3d2e8b93e4d972af6195be696a5ea133e"
Sep 09 21:09:12.707720 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:12.707697    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"]
Sep 09 21:09:12.711542 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:12.711518    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-98dad6/test-model-98dad6-kserve-7795bc85-pml29"]
Sep 09 21:09:13.845962 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:13.845926    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a51d8c-e4d7-4024-95ea-61d42d8ddebe" path="/var/lib/kubelet/pods/20a51d8c-e4d7-4024-95ea-61d42d8ddebe/volumes"
Sep 09 21:09:14.671881 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:14.671844    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:16.011358 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:16.011320    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:09:19.677344 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:19.677298    2578 patch_prober.go:28] interesting pod/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 21:09:19.677728 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:19.677384    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:09:20.725954 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:20.725918    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerID="c6445131e6d9e19a052c8afc61b041ee8e2ba72261eac6d6992b42dd03729dd5" exitCode=0
Sep 09 21:09:20.726502 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:20.725995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" event={"ID":"1ee0819d-2938-41fd-abb7-1f456fc29753","Type":"ContainerDied","Data":"c6445131e6d9e19a052c8afc61b041ee8e2ba72261eac6d6992b42dd03729dd5"}
Sep 09 21:09:20.726502 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:20.726490    2578 scope.go:117] "RemoveContainer" containerID="c6445131e6d9e19a052c8afc61b041ee8e2ba72261eac6d6992b42dd03729dd5"
Sep 09 21:09:20.727499 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:20.727472    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:09:20.727598 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:20.727535    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:09:20.727598 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:20.727576    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:09:21.736904 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:21.736859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" event={"ID":"1ee0819d-2938-41fd-abb7-1f456fc29753","Type":"ContainerStarted","Data":"a1f575d8bd35cf70fdc0c6d25e84b7da12e1c78c161cf2a62bc31a99ff73ac27"}
Sep 09 21:09:22.567232 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:22.567167    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:22.567469 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:22.567441    2578 patch_prober.go:28] interesting pod/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 21:09:22.567529 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:22.567509    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 21:09:23.567612 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:23.567574    2578 patch_prober.go:28] interesting pod/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 21:09:23.568023 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:23.567641    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 21:09:24.568090 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:24.568052    2578 patch_prober.go:28] interesting pod/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 21:09:24.568498 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:24.568121    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 21:09:25.567125 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:25.567087    2578 patch_prober.go:28] interesting pod/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 21:09:25.567318 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:25.567144    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 21:09:26.567683 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:26.567650    2578 patch_prober.go:28] interesting pod/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 21:09:26.568054 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:26.567703    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 21:09:27.570615 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:27.570584    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:27.571080 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:27.570972    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:27.571594 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:27.571574    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:09:27.910241 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:27.910206    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:09:32.674011 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:32.673976    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"]
Sep 09 21:09:32.674568 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:32.674241    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy" containerID="cri-o://f32808716fa01a069019ec7eb69d4d456eb56e6c8d7111333cf50c38fcb96b85" gracePeriod=30
Sep 09 21:09:37.814902 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.814864    2578 generic.go:356] "Generic (PLEG): container finished" podID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerID="f32808716fa01a069019ec7eb69d4d456eb56e6c8d7111333cf50c38fcb96b85" exitCode=0
Sep 09 21:09:37.814902 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.814893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" event={"ID":"dc426cc6-f400-4bce-a079-a223ae4aad8c","Type":"ContainerDied","Data":"f32808716fa01a069019ec7eb69d4d456eb56e6c8d7111333cf50c38fcb96b85"}
Sep 09 21:09:37.815430 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.814962    2578 scope.go:117] "RemoveContainer" containerID="3b707f24d8ec46a5ec7e42fc36eb29512165dacabc7a6aa4da127955b9e1b506"
Sep 09 21:09:37.930896 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.930873    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:09:37.947894 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.947847    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc426cc6-f400-4bce-a079-a223ae4aad8c-istiod-ca-cert\") pod \"dc426cc6-f400-4bce-a079-a223ae4aad8c\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") "
Sep 09 21:09:37.947963 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.947893    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-credential-socket\") pod \"dc426cc6-f400-4bce-a079-a223ae4aad8c\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") "
Sep 09 21:09:37.947963 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.947932    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-envoy\") pod \"dc426cc6-f400-4bce-a079-a223ae4aad8c\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") "
Sep 09 21:09:37.948050 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.947963    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-podinfo\") pod \"dc426cc6-f400-4bce-a079-a223ae4aad8c\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") "
Sep 09 21:09:37.948050 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.947995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-data\") pod \"dc426cc6-f400-4bce-a079-a223ae4aad8c\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") "
Sep 09 21:09:37.948050 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.948019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpxtg\" (UniqueName: \"kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-kube-api-access-lpxtg\") pod \"dc426cc6-f400-4bce-a079-a223ae4aad8c\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") "
Sep 09 21:09:37.948050 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.948046    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-socket\") pod \"dc426cc6-f400-4bce-a079-a223ae4aad8c\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") "
Sep 09 21:09:37.948278 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.948095    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-certs\") pod \"dc426cc6-f400-4bce-a079-a223ae4aad8c\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") "
Sep 09 21:09:37.948278 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.948148    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-token\") pod \"dc426cc6-f400-4bce-a079-a223ae4aad8c\" (UID: \"dc426cc6-f400-4bce-a079-a223ae4aad8c\") "
Sep 09 21:09:37.948278 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.948229    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc426cc6-f400-4bce-a079-a223ae4aad8c" (UID: "dc426cc6-f400-4bce-a079-a223ae4aad8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:37.948278 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.948229    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc426cc6-f400-4bce-a079-a223ae4aad8c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc426cc6-f400-4bce-a079-a223ae4aad8c" (UID: "dc426cc6-f400-4bce-a079-a223ae4aad8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:37.948485 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.948387    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc426cc6-f400-4bce-a079-a223ae4aad8c" (UID: "dc426cc6-f400-4bce-a079-a223ae4aad8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:37.948485 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.948397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc426cc6-f400-4bce-a079-a223ae4aad8c" (UID: "dc426cc6-f400-4bce-a079-a223ae4aad8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:37.948485 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.948445    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc426cc6-f400-4bce-a079-a223ae4aad8c-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:37.948485 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.948462    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:37.948794 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.948752    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc426cc6-f400-4bce-a079-a223ae4aad8c" (UID: "dc426cc6-f400-4bce-a079-a223ae4aad8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:37.950327 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.950300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc426cc6-f400-4bce-a079-a223ae4aad8c" (UID: "dc426cc6-f400-4bce-a079-a223ae4aad8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:37.950460 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.950434    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc426cc6-f400-4bce-a079-a223ae4aad8c" (UID: "dc426cc6-f400-4bce-a079-a223ae4aad8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:09:37.950709 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.950680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-kube-api-access-lpxtg" (OuterVolumeSpecName: "kube-api-access-lpxtg") pod "dc426cc6-f400-4bce-a079-a223ae4aad8c" (UID: "dc426cc6-f400-4bce-a079-a223ae4aad8c"). InnerVolumeSpecName "kube-api-access-lpxtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:37.950796 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:37.950775    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc426cc6-f400-4bce-a079-a223ae4aad8c" (UID: "dc426cc6-f400-4bce-a079-a223ae4aad8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:38.049195 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.049136    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:38.049195 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.049188    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:38.049195 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.049198    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:38.049434 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.049206    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpxtg\" (UniqueName: \"kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-kube-api-access-lpxtg\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:38.049434 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.049217    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:38.049434 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.049225    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc426cc6-f400-4bce-a079-a223ae4aad8c-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:38.049434 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.049234    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc426cc6-f400-4bce-a079-a223ae4aad8c-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:38.825916 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.825887    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"
Sep 09 21:09:38.826414 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.825883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q" event={"ID":"dc426cc6-f400-4bce-a079-a223ae4aad8c","Type":"ContainerDied","Data":"f1ad430187efa2bb7306054834c7f84bcbdaa8bdaaec9e166e3d83200dfec215"}
Sep 09 21:09:38.826414 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.826039    2578 scope.go:117] "RemoveContainer" containerID="f32808716fa01a069019ec7eb69d4d456eb56e6c8d7111333cf50c38fcb96b85"
Sep 09 21:09:38.856594 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.856567    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"]
Sep 09 21:09:38.861010 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:38.860982    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-98dad6-openshift-default-79764cf5b-tlh8q"]
Sep 09 21:09:39.840669 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:39.840638    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" path="/var/lib/kubelet/pods/dc426cc6-f400-4bce-a079-a223ae4aad8c/volumes"
Sep 09 21:09:41.110557 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:41.110520    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:09:50.029950 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.029913    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"]
Sep 09 21:09:50.030420 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.030188    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt" podUID="ce33ff71-1173-498a-9ac7-2fba603bda5b" containerName="maas-api" containerID="cri-o://e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6" gracePeriod=30
Sep 09 21:09:50.270412 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.270388    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:50.355048 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.355023    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-maas-api-tls\") pod \"ce33ff71-1173-498a-9ac7-2fba603bda5b\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") "
Sep 09 21:09:50.355214 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.355073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlhm5\" (UniqueName: \"kubernetes.io/projected/ce33ff71-1173-498a-9ac7-2fba603bda5b-kube-api-access-tlhm5\") pod \"ce33ff71-1173-498a-9ac7-2fba603bda5b\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") "
Sep 09 21:09:50.355276 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.355253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-metrics-certs\") pod \"ce33ff71-1173-498a-9ac7-2fba603bda5b\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") "
Sep 09 21:09:50.355319 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.355309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce33ff71-1173-498a-9ac7-2fba603bda5b-openshift-service-ca\") pod \"ce33ff71-1173-498a-9ac7-2fba603bda5b\" (UID: \"ce33ff71-1173-498a-9ac7-2fba603bda5b\") "
Sep 09 21:09:50.355667 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.355642    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce33ff71-1173-498a-9ac7-2fba603bda5b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ce33ff71-1173-498a-9ac7-2fba603bda5b" (UID: "ce33ff71-1173-498a-9ac7-2fba603bda5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:50.357189 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.357151    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ce33ff71-1173-498a-9ac7-2fba603bda5b" (UID: "ce33ff71-1173-498a-9ac7-2fba603bda5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:50.357278 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.357211    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce33ff71-1173-498a-9ac7-2fba603bda5b-kube-api-access-tlhm5" (OuterVolumeSpecName: "kube-api-access-tlhm5") pod "ce33ff71-1173-498a-9ac7-2fba603bda5b" (UID: "ce33ff71-1173-498a-9ac7-2fba603bda5b"). InnerVolumeSpecName "kube-api-access-tlhm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:50.357325 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.357268    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ce33ff71-1173-498a-9ac7-2fba603bda5b" (UID: "ce33ff71-1173-498a-9ac7-2fba603bda5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:50.456474 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.456442    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:50.456474 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.456468    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce33ff71-1173-498a-9ac7-2fba603bda5b-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:50.456474 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.456478    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce33ff71-1173-498a-9ac7-2fba603bda5b-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:50.456705 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.456486    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlhm5\" (UniqueName: \"kubernetes.io/projected/ce33ff71-1173-498a-9ac7-2fba603bda5b-kube-api-access-tlhm5\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:09:50.878295 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.878265    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce33ff71-1173-498a-9ac7-2fba603bda5b" containerID="e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6" exitCode=0
Sep 09 21:09:50.878478 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.878303    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"
Sep 09 21:09:50.878478 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.878339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt" event={"ID":"ce33ff71-1173-498a-9ac7-2fba603bda5b","Type":"ContainerDied","Data":"e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6"}
Sep 09 21:09:50.878478 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.878385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt" event={"ID":"ce33ff71-1173-498a-9ac7-2fba603bda5b","Type":"ContainerDied","Data":"105a4e8d89d1cf4804ee9d2c66b24caee073cbdc1e7e1386f998eda253a818c3"}
Sep 09 21:09:50.878478 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.878406    2578 scope.go:117] "RemoveContainer" containerID="e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6"
Sep 09 21:09:50.886894 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.886876    2578 scope.go:117] "RemoveContainer" containerID="e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6"
Sep 09 21:09:50.887139 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:09:50.887121    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6\": container with ID starting with e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6 not found: ID does not exist" containerID="e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6"
Sep 09 21:09:50.887207 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.887147    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6"} err="failed to get container status \"e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6\": rpc error: code = NotFound desc = could not find container \"e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6\": container with ID starting with e30fc2acc4bbecdd118c73c612bf33920937e78feff53c12c4bb8fed729f7eb6 not found: ID does not exist"
Sep 09 21:09:50.899919 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.899893    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"]
Sep 09 21:09:50.904898 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:50.904879    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57783aa9-556648d477-kptdt"]
Sep 09 21:09:51.840550 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:51.840513    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce33ff71-1173-498a-9ac7-2fba603bda5b" path="/var/lib/kubelet/pods/ce33ff71-1173-498a-9ac7-2fba603bda5b/volumes"
Sep 09 21:09:52.524664 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:52.524627    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:09:56.720839 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:56.720802    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"]
Sep 09 21:09:56.721314 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:56.721014    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy" containerID="cri-o://a1f575d8bd35cf70fdc0c6d25e84b7da12e1c78c161cf2a62bc31a99ff73ac27" gracePeriod=30
Sep 09 21:09:57.573782 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:57.573748    2578 patch_prober.go:28] interesting pod/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 21:09:57.573943 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:09:57.573812    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:10:01.930074 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:01.930044    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerID="a1f575d8bd35cf70fdc0c6d25e84b7da12e1c78c161cf2a62bc31a99ff73ac27" exitCode=0
Sep 09 21:10:01.930467 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:01.930115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" event={"ID":"1ee0819d-2938-41fd-abb7-1f456fc29753","Type":"ContainerDied","Data":"a1f575d8bd35cf70fdc0c6d25e84b7da12e1c78c161cf2a62bc31a99ff73ac27"}
Sep 09 21:10:01.930467 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:01.930193    2578 scope.go:117] "RemoveContainer" containerID="c6445131e6d9e19a052c8afc61b041ee8e2ba72261eac6d6992b42dd03729dd5"
Sep 09 21:10:01.965204 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:01.965167    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:10:02.049202 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049079    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-certs\") pod \"1ee0819d-2938-41fd-abb7-1f456fc29753\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") "
Sep 09 21:10:02.049202 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-envoy\") pod \"1ee0819d-2938-41fd-abb7-1f456fc29753\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") "
Sep 09 21:10:02.049202 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049142    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhz8t\" (UniqueName: \"kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-kube-api-access-nhz8t\") pod \"1ee0819d-2938-41fd-abb7-1f456fc29753\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") "
Sep 09 21:10:02.049202 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049157    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-token\") pod \"1ee0819d-2938-41fd-abb7-1f456fc29753\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") "
Sep 09 21:10:02.049202 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049196    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-socket\") pod \"1ee0819d-2938-41fd-abb7-1f456fc29753\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") "
Sep 09 21:10:02.049570 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049402    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-data\") pod \"1ee0819d-2938-41fd-abb7-1f456fc29753\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") "
Sep 09 21:10:02.049570 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049437    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ee0819d-2938-41fd-abb7-1f456fc29753" (UID: "1ee0819d-2938-41fd-abb7-1f456fc29753"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:02.049570 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049484    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-credential-socket\") pod \"1ee0819d-2938-41fd-abb7-1f456fc29753\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") "
Sep 09 21:10:02.049570 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049503    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ee0819d-2938-41fd-abb7-1f456fc29753" (UID: "1ee0819d-2938-41fd-abb7-1f456fc29753"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:02.049570 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049520    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-podinfo\") pod \"1ee0819d-2938-41fd-abb7-1f456fc29753\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") "
Sep 09 21:10:02.049831 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee0819d-2938-41fd-abb7-1f456fc29753-istiod-ca-cert\") pod \"1ee0819d-2938-41fd-abb7-1f456fc29753\" (UID: \"1ee0819d-2938-41fd-abb7-1f456fc29753\") "
Sep 09 21:10:02.049831 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049588    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ee0819d-2938-41fd-abb7-1f456fc29753" (UID: "1ee0819d-2938-41fd-abb7-1f456fc29753"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:02.049831 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049718    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ee0819d-2938-41fd-abb7-1f456fc29753" (UID: "1ee0819d-2938-41fd-abb7-1f456fc29753"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:02.050002 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049857    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:10:02.050002 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049876    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:10:02.050002 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049893    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:10:02.050002 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049907    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:10:02.050002 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.049920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee0819d-2938-41fd-abb7-1f456fc29753-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ee0819d-2938-41fd-abb7-1f456fc29753" (UID: "1ee0819d-2938-41fd-abb7-1f456fc29753"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:02.051426 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.051403    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-kube-api-access-nhz8t" (OuterVolumeSpecName: "kube-api-access-nhz8t") pod "1ee0819d-2938-41fd-abb7-1f456fc29753" (UID: "1ee0819d-2938-41fd-abb7-1f456fc29753"). InnerVolumeSpecName "kube-api-access-nhz8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:02.051538 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.051512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ee0819d-2938-41fd-abb7-1f456fc29753" (UID: "1ee0819d-2938-41fd-abb7-1f456fc29753"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:02.051802 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.051778    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ee0819d-2938-41fd-abb7-1f456fc29753" (UID: "1ee0819d-2938-41fd-abb7-1f456fc29753"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:10:02.051802 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.051788    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ee0819d-2938-41fd-abb7-1f456fc29753" (UID: "1ee0819d-2938-41fd-abb7-1f456fc29753"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:02.150412 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.150377    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:10:02.150412 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.150405    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee0819d-2938-41fd-abb7-1f456fc29753-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:10:02.150412 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.150416    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:10:02.150658 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.150425    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhz8t\" (UniqueName: \"kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-kube-api-access-nhz8t\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:10:02.150658 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.150435    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee0819d-2938-41fd-abb7-1f456fc29753-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:10:02.935901 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.935864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82" event={"ID":"1ee0819d-2938-41fd-abb7-1f456fc29753","Type":"ContainerDied","Data":"e33c1ec89c607a5aae8bf3d00d163df68486077ba394aa0529075c9a7ccf6322"}
Sep 09 21:10:02.935901 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.935881    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"
Sep 09 21:10:02.936391 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.935922    2578 scope.go:117] "RemoveContainer" containerID="a1f575d8bd35cf70fdc0c6d25e84b7da12e1c78c161cf2a62bc31a99ff73ac27"
Sep 09 21:10:02.955812 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.955789    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"]
Sep 09 21:10:02.960202 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:02.960158    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-57783aa9-openshift-default-5b87668db4-w5g82"]
Sep 09 21:10:03.840308 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:03.840269    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" path="/var/lib/kubelet/pods/1ee0819d-2938-41fd-abb7-1f456fc29753/volumes"
Sep 09 21:10:07.203300 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203267    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"]
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203558    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203569    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203580    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df08a066-11dc-43df-8245-c9906f780c93" containerName="main"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203585    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="df08a066-11dc-43df-8245-c9906f780c93" containerName="main"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203597    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce33ff71-1173-498a-9ac7-2fba603bda5b" containerName="maas-api"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203602    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce33ff71-1173-498a-9ac7-2fba603bda5b" containerName="maas-api"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203611    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203616    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203621    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203626    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203635    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203640    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203648    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a51d8c-e4d7-4024-95ea-61d42d8ddebe" containerName="main"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203653    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a51d8c-e4d7-4024-95ea-61d42d8ddebe" containerName="main"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203697    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="df08a066-11dc-43df-8245-c9906f780c93" containerName="main"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203704    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203710    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce33ff71-1173-498a-9ac7-2fba603bda5b" containerName="maas-api"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203717    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ee0819d-2938-41fd-abb7-1f456fc29753" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203737    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203742    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc426cc6-f400-4bce-a079-a223ae4aad8c" containerName="istio-proxy"
Sep 09 21:10:07.203767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.203748    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a51d8c-e4d7-4024-95ea-61d42d8ddebe" containerName="main"
Sep 09 21:10:07.206637 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.206610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.208904 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.208882    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8db5a7-openshift-default-dockercfg-k6bbn\""
Sep 09 21:10:07.218283 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.218263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"]
Sep 09 21:10:07.296738 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.296706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-certs\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.296738 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.296745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-data\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.296997 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.296794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtpb\" (UniqueName: \"kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-kube-api-access-qxtpb\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.296997 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.296824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-podinfo\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.296997 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.296859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-socket\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.296997 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.296893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-token\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.296997 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.296930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-credential-socket\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.296997 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.296974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-envoy\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.297273 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.297054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22feda42-4b1e-40b5-8541-71ae2b1309c0-istiod-ca-cert\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398089 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtpb\" (UniqueName: \"kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-kube-api-access-qxtpb\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398089 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-podinfo\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-socket\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-token\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-credential-socket\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-envoy\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22feda42-4b1e-40b5-8541-71ae2b1309c0-istiod-ca-cert\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-certs\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-data\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398718 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-socket\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398718 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398672    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-data\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398825 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-credential-socket\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398884 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-certs\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.398993 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.398973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22feda42-4b1e-40b5-8541-71ae2b1309c0-istiod-ca-cert\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.400494 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.400470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-envoy\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.400596 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.400568    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-podinfo\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.405631 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.405606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-token\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.405631 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.405621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtpb\" (UniqueName: \"kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-kube-api-access-qxtpb\") pod \"e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") " pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.516726 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.516641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:07.789932 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.789847    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"]
Sep 09 21:10:07.796679 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.796642    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:07.796780 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.796713    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:07.796780 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.796742    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:07.961552 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.961513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" event={"ID":"22feda42-4b1e-40b5-8541-71ae2b1309c0","Type":"ContainerStarted","Data":"610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a"}
Sep 09 21:10:07.961736 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:07.961573    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" event={"ID":"22feda42-4b1e-40b5-8541-71ae2b1309c0","Type":"ContainerStarted","Data":"81ba6bcc260530007fff35018dbc17578d119e37bbffe83f440bc76d23a26820"}
Sep 09 21:10:08.037107 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:08.037048    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" podStartSLOduration=1.037032063 podStartE2EDuration="1.037032063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:08.036460083 +0000 UTC m=+1284.691240240" watchObservedRunningTime="2026-09-09 21:10:08.037032063 +0000 UTC m=+1284.691812215"
Sep 09 21:10:08.517699 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:08.517662    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:08.522503 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:08.522477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:08.966345 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:08.966312    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:08.967345 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:08.967326    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:11.056773 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.056742    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"]
Sep 09 21:10:11.058910 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.058894    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.061229 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.061200    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-629537-openshift-default-dockercfg-rwcxw\""
Sep 09 21:10:11.074069 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.074050    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"]
Sep 09 21:10:11.231591 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.231553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-envoy\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.231591 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.231596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-certs\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.231839 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.231665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-data\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.231839 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.231703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-podinfo\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.231839 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.231755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-credential-socket\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.231839 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.231801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-token\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.231839 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.231827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-socket\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.232026 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.231850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwch\" (UniqueName: \"kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-kube-api-access-hzwch\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.232026 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.231873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4262be42-a2d0-4cce-a596-26c57f2aa11f-istiod-ca-cert\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.305878 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.305845    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"]
Sep 09 21:10:11.308347 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.308286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.310550 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.310529    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-gateway-identity\""
Sep 09 21:10:11.310821 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.310797    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 21:10:11.311050 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.311023    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:10:11.311389 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.311367    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-8db5a7-serving-cert\""
Sep 09 21:10:11.311488 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.311369    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 21:10:11.311554 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.311518    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gj4rp\""
Sep 09 21:10:11.324125 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.324098    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"]
Sep 09 21:10:11.333060 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-envoy\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333209 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-certs\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333209 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-data\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333209 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-podinfo\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333209 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-credential-socket\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333209 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-token\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333507 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-socket\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333507 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwch\" (UniqueName: \"kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-kube-api-access-hzwch\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333507 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4262be42-a2d0-4cce-a596-26c57f2aa11f-istiod-ca-cert\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333507 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-data\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333727 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-credential-socket\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333786 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-socket\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.333843 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.333810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-certs\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.334133 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.334110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4262be42-a2d0-4cce-a596-26c57f2aa11f-istiod-ca-cert\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.335447 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.335419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-envoy\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.335820 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.335796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-podinfo\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.344967 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.344947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwch\" (UniqueName: \"kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-kube-api-access-hzwch\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.345164 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.345145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-token\") pod \"e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") " pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.370203 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.370159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:11.434251 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.434113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.434251 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.434161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/3ea85657-996d-4ca4-98fd-a01edc768074-kube-api-access-5th4k\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.434717 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.434605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea85657-996d-4ca4-98fd-a01edc768074-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.434717 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.434650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.535308 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.535276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.535482 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.535316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/3ea85657-996d-4ca4-98fd-a01edc768074-kube-api-access-5th4k\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.535482 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.535384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea85657-996d-4ca4-98fd-a01edc768074-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.535482 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.535410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.535660 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:10:11.535505    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-serving-cert: secret "maas-api-e2e-shared-b-w3-8db5a7-serving-cert" not found
Sep 09 21:10:11.535660 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:10:11.535575    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-maas-api-tls podName:3ea85657-996d-4ca4-98fd-a01edc768074 nodeName:}" failed. No retries permitted until 2026-09-09 21:10:12.03555428 +0000 UTC m=+1288.690334437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-maas-api-tls") pod "maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6" (UID: "3ea85657-996d-4ca4-98fd-a01edc768074") : secret "maas-api-e2e-shared-b-w3-8db5a7-serving-cert" not found
Sep 09 21:10:11.536083 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.536063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea85657-996d-4ca4-98fd-a01edc768074-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.537707 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.537687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.543770 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.543751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/3ea85657-996d-4ca4-98fd-a01edc768074-kube-api-access-5th4k\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:11.650143 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.650118    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"]
Sep 09 21:10:11.652709 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:10:11.652673    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4262be42_a2d0_4cce_a596_26c57f2aa11f.slice/crio-d721fc00053d3562cfc58946052d1e50a8be31a38bb7d4835f7815eb40864de0 WatchSource:0}: Error finding container d721fc00053d3562cfc58946052d1e50a8be31a38bb7d4835f7815eb40864de0: Status 404 returned error can't find the container with id d721fc00053d3562cfc58946052d1e50a8be31a38bb7d4835f7815eb40864de0
Sep 09 21:10:11.655433 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.655397    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:11.655550 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.655493    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:11.655550 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.655538    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:11.983457 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.983361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" event={"ID":"4262be42-a2d0-4cce-a596-26c57f2aa11f","Type":"ContainerStarted","Data":"5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19"}
Sep 09 21:10:11.983457 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:11.983418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" event={"ID":"4262be42-a2d0-4cce-a596-26c57f2aa11f","Type":"ContainerStarted","Data":"d721fc00053d3562cfc58946052d1e50a8be31a38bb7d4835f7815eb40864de0"}
Sep 09 21:10:12.003766 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:12.003718    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" podStartSLOduration=1.00370432 podStartE2EDuration="1.00370432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:12.001971917 +0000 UTC m=+1288.656752077" watchObservedRunningTime="2026-09-09 21:10:12.00370432 +0000 UTC m=+1288.658484478"
Sep 09 21:10:12.039100 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:12.039060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:12.041498 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:12.041461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:12.225936 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:12.225893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:12.370829 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:12.370796    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:12.375670 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:12.375647    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:12.492260 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:12.492235    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"]
Sep 09 21:10:12.494016 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:10:12.493993    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea85657_996d_4ca4_98fd_a01edc768074.slice/crio-3c3fe454af47688d81d40a657dc5daf56d80d01b9a1ed56ba69ec22c1039c1de WatchSource:0}: Error finding container 3c3fe454af47688d81d40a657dc5daf56d80d01b9a1ed56ba69ec22c1039c1de: Status 404 returned error can't find the container with id 3c3fe454af47688d81d40a657dc5daf56d80d01b9a1ed56ba69ec22c1039c1de
Sep 09 21:10:12.989063 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:12.989018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6" event={"ID":"3ea85657-996d-4ca4-98fd-a01edc768074","Type":"ContainerStarted","Data":"a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3"}
Sep 09 21:10:12.989265 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:12.989089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6" event={"ID":"3ea85657-996d-4ca4-98fd-a01edc768074","Type":"ContainerStarted","Data":"3c3fe454af47688d81d40a657dc5daf56d80d01b9a1ed56ba69ec22c1039c1de"}
Sep 09 21:10:12.989328 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:12.989291    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:12.990274 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:12.990252    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:13.007402 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:13.007357    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6" podStartSLOduration=1.653177334 podStartE2EDuration="2.0073423s" totalImagesPullingTime="354.164966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:13.005814728 +0000 UTC m=+1289.660594887" watchObservedRunningTime="2026-09-09 21:10:13.0073423 +0000 UTC m=+1289.662122456"
Sep 09 21:10:13.994250 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:13.994214    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:15.318046 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.318013    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"]
Sep 09 21:10:15.320618 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.320598    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.324278 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.324210    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-629537-serving-cert\""
Sep 09 21:10:15.341276 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.341251    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"]
Sep 09 21:10:15.473032 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.472995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvlf\" (UniqueName: \"kubernetes.io/projected/fa365176-6a0a-4597-bfe7-7bd70d48abaf-kube-api-access-2dvlf\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.473032 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.473037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.473269 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.473071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa365176-6a0a-4597-bfe7-7bd70d48abaf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.473269 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.473163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.573748 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.573660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.573890 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.573774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvlf\" (UniqueName: \"kubernetes.io/projected/fa365176-6a0a-4597-bfe7-7bd70d48abaf-kube-api-access-2dvlf\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.573890 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.573808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.573890 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.573840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa365176-6a0a-4597-bfe7-7bd70d48abaf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.574590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.574563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa365176-6a0a-4597-bfe7-7bd70d48abaf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.576278 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.576258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.576609 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.576587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.582156 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.582125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvlf\" (UniqueName: \"kubernetes.io/projected/fa365176-6a0a-4597-bfe7-7bd70d48abaf-kube-api-access-2dvlf\") pod \"maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.631908 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.631881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:15.906902 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:15.906878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"]
Sep 09 21:10:15.908602 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:10:15.908573    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa365176_6a0a_4597_bfe7_7bd70d48abaf.slice/crio-f1c6389e2f98bd7d02fcc4b0a2287b3a7a9189e28ee92d6b63b288864ce2e078 WatchSource:0}: Error finding container f1c6389e2f98bd7d02fcc4b0a2287b3a7a9189e28ee92d6b63b288864ce2e078: Status 404 returned error can't find the container with id f1c6389e2f98bd7d02fcc4b0a2287b3a7a9189e28ee92d6b63b288864ce2e078
Sep 09 21:10:16.004846 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:16.004813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz" event={"ID":"fa365176-6a0a-4597-bfe7-7bd70d48abaf","Type":"ContainerStarted","Data":"f1c6389e2f98bd7d02fcc4b0a2287b3a7a9189e28ee92d6b63b288864ce2e078"}
Sep 09 21:10:16.033818 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:16.033781    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:10:17.010445 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:17.010401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz" event={"ID":"fa365176-6a0a-4597-bfe7-7bd70d48abaf","Type":"ContainerStarted","Data":"22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa"}
Sep 09 21:10:17.010919 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:17.010531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:20.004319 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:20.004290    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:10:20.022349 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:20.022288    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz" podStartSLOduration=4.480034933 podStartE2EDuration="5.02226772s" totalImagesPullingTime="542.232787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:17.041133509 +0000 UTC m=+1293.695913667" watchObservedRunningTime="2026-09-09 21:10:20.02226772 +0000 UTC m=+1296.677047879"
Sep 09 21:10:23.033332 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:23.032441    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:10:25.008315 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:25.008277    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 21:10:25.008691 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:25.008334    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:10:25.105817 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:25.105780    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:10:27.077787 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:27.077762    2578 generic.go:356] "Generic (PLEG): container finished" podID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerID="610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a" exitCode=0
Sep 09 21:10:27.078475 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:27.077803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" event={"ID":"22feda42-4b1e-40b5-8541-71ae2b1309c0","Type":"ContainerDied","Data":"610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a"}
Sep 09 21:10:27.078475 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:27.078152    2578 scope.go:117] "RemoveContainer" containerID="610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a"
Sep 09 21:10:27.079187 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:27.079140    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:27.079292 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:27.079242    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:27.079355 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:27.079291    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:28.084124 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:28.084084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" event={"ID":"22feda42-4b1e-40b5-8541-71ae2b1309c0","Type":"ContainerStarted","Data":"2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5"}
Sep 09 21:10:28.517993 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:28.517896    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:28.518704 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:28.518274    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 09 21:10:28.518704 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:28.518428    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 09 21:10:29.031440 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:29.031394    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 21:10:29.031656 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:29.031464    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:10:29.518280 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:29.517804    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 09 21:10:29.518280 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:29.517866    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 09 21:10:30.517528 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:30.517495    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 09 21:10:30.518132 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:30.517685    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 09 21:10:31.099136 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:31.099100    2578 generic.go:356] "Generic (PLEG): container finished" podID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerID="5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19" exitCode=0
Sep 09 21:10:31.099524 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:31.099191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" event={"ID":"4262be42-a2d0-4cce-a596-26c57f2aa11f","Type":"ContainerDied","Data":"5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19"}
Sep 09 21:10:31.099573 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:31.099563    2578 scope.go:117] "RemoveContainer" containerID="5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19"
Sep 09 21:10:31.100643 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:31.100614    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:31.100703 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:31.100689    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:31.100740 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:31.100729    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:31.517212 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:31.517099    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 09 21:10:31.517212 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:31.517190    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 09 21:10:32.109324 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:32.109290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" event={"ID":"4262be42-a2d0-4cce-a596-26c57f2aa11f","Type":"ContainerStarted","Data":"c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176"}
Sep 09 21:10:32.371353 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:32.371257    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:32.371507 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:32.371419    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 21:10:32.371507 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:32.371465    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 21:10:32.517136 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:32.517089    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 09 21:10:32.517366 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:32.517163    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 09 21:10:33.371341 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:33.371297    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 21:10:33.371736 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:33.371369    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 21:10:33.527248 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:33.527213    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:33.528378 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:33.528351    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:33.530815 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:33.530775    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:10:34.371326 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:34.371285    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 21:10:34.371510 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:34.371349    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 21:10:35.371439 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:35.371401    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 21:10:35.371626 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:35.371459    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 21:10:36.370604 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:36.370564    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 21:10:36.370796 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:36.370628    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 21:10:37.371492 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:37.371451    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 21:10:37.371881 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:37.371508    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 21:10:38.403051 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:38.403020    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:38.403452 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:38.403432    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:38.404217 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:38.404195    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:10:39.543126 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.543093    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"]
Sep 09 21:10:39.545722 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.545701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.548265 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.548246    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-459158-openshift-default-dockercfg-hzh8t\""
Sep 09 21:10:39.557943 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.557921    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"]
Sep 09 21:10:39.620372 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.620341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istiod-ca-cert\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.620567 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.620380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-envoy\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.620567 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.620403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-token\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.620567 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.620464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-credential-socket\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.620567 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.620512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-socket\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.620778 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.620590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-data\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.620778 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.620641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49kfq\" (UniqueName: \"kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-kube-api-access-49kfq\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.620778 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.620711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-certs\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.620778 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.620762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-podinfo\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722120 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-certs\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722120 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-podinfo\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722414 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istiod-ca-cert\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722414 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-envoy\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722414 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-token\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722414 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-credential-socket\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722414 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-socket\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722679 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-data\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722679 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49kfq\" (UniqueName: \"kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-kube-api-access-49kfq\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722679 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-certs\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722842 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-credential-socket\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.722946 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-data\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.723004 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722946    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-socket\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.723004 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.722950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istiod-ca-cert\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.725038 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.725016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-envoy\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.725147 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.725133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-podinfo\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.730446 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.730422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-token\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.730640 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.730608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49kfq\" (UniqueName: \"kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-kube-api-access-49kfq\") pod \"e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") " pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:39.855701 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:39.855671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:40.132580 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:40.132554    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"]
Sep 09 21:10:40.135105 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:10:40.135068    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac18b9ad_1250_4a5f_8ec9_00234524b7f8.slice/crio-957f0f4dbd0f4521169cf143cb7fbc8ed1f2b60818ccd06542fcecdb046d5555 WatchSource:0}: Error finding container 957f0f4dbd0f4521169cf143cb7fbc8ed1f2b60818ccd06542fcecdb046d5555: Status 404 returned error can't find the container with id 957f0f4dbd0f4521169cf143cb7fbc8ed1f2b60818ccd06542fcecdb046d5555
Sep 09 21:10:40.137246 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:40.137213    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:40.137346 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:40.137280    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:40.137346 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:40.137307    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:10:40.156618 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:40.156594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" event={"ID":"ac18b9ad-1250-4a5f-8ec9-00234524b7f8","Type":"ContainerStarted","Data":"957f0f4dbd0f4521169cf143cb7fbc8ed1f2b60818ccd06542fcecdb046d5555"}
Sep 09 21:10:41.162942 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:41.162904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" event={"ID":"ac18b9ad-1250-4a5f-8ec9-00234524b7f8","Type":"ContainerStarted","Data":"685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e"}
Sep 09 21:10:41.182669 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:41.182621    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" podStartSLOduration=2.182605082 podStartE2EDuration="2.182605082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:41.181294836 +0000 UTC m=+1317.836075009" watchObservedRunningTime="2026-09-09 21:10:41.182605082 +0000 UTC m=+1317.837385239"
Sep 09 21:10:41.856781 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:41.856753    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:41.861514 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:41.861490    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:42.167708 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:42.167623    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:42.168544 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:42.168525    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:10:50.809851 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:10:50.809822    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:11:01.264295 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:01.264206    2578 generic.go:356] "Generic (PLEG): container finished" podID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerID="685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e" exitCode=0
Sep 09 21:11:01.264295 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:01.264256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" event={"ID":"ac18b9ad-1250-4a5f-8ec9-00234524b7f8","Type":"ContainerDied","Data":"685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e"}
Sep 09 21:11:01.264750 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:01.264648    2578 scope.go:117] "RemoveContainer" containerID="685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e"
Sep 09 21:11:01.265760 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:01.265731    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:11:01.265817 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:01.265801    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:11:01.265857 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:01.265831    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:11:02.204642 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:02.204595    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:11:02.270960 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:02.270924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" event={"ID":"ac18b9ad-1250-4a5f-8ec9-00234524b7f8","Type":"ContainerStarted","Data":"2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5"}
Sep 09 21:11:02.856399 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:02.856356    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:11:02.856644 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:02.856621    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 21:11:02.856691 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:02.856671    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 21:11:03.856472 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:03.856434    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 21:11:03.856929 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:03.856493    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 21:11:04.856093 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:04.856056    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 21:11:04.856294 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:04.856115    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 21:11:05.855989 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:05.855952    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 21:11:05.856520 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:05.856020    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 21:11:06.856977 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:06.856938    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 21:11:06.857426 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:06.857022    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 21:11:07.934106 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:07.934070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:11:07.934512 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:07.934393    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:11:07.935128 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:07.935081    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:11:17.585604 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.585570    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"]
Sep 09 21:11:17.588071 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.588052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.590915 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.590893    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f8b5626b-openshift-default-dockercfg-glsr6\""
Sep 09 21:11:17.602917 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.602895    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"]
Sep 09 21:11:17.644503 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.644474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b06e898-48d3-4066-9224-428d5972d978-istiod-ca-cert\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.644503 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.644506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-certs\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.644662 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.644589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-credential-socket\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.644662 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.644620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-data\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.644662 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.644640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b06e898-48d3-4066-9224-428d5972d978-istio-podinfo\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.644662 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.644658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9x4\" (UniqueName: \"kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-kube-api-access-rm9x4\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.644793 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.644691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-socket\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.644793 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.644711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-envoy\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.644793 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.644726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-istio-token\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.745738 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.745706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-socket\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.745738 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.745737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-envoy\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.745995 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.745753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-istio-token\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.745995 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.745791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b06e898-48d3-4066-9224-428d5972d978-istiod-ca-cert\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.745995 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.745812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-certs\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.745995 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.745893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-credential-socket\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.745995 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.745924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-data\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.745995 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.745954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b06e898-48d3-4066-9224-428d5972d978-istio-podinfo\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.745995 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.745981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9x4\" (UniqueName: \"kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-kube-api-access-rm9x4\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.746391 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.746221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-socket\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.746391 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.746345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-data\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.746391 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.746378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-certs\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.746521 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.746429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-credential-socket\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.746579 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.746556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b06e898-48d3-4066-9224-428d5972d978-istiod-ca-cert\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.748340 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.748313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-envoy\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.748442 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.748403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b06e898-48d3-4066-9224-428d5972d978-istio-podinfo\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.754780 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.754759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-istio-token\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.758489 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.758472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9x4\" (UniqueName: \"kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-kube-api-access-rm9x4\") pod \"e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") " pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:17.898119 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:17.898038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:18.189619 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:18.189591    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"]
Sep 09 21:11:18.190023 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:11:18.189987    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b06e898_48d3_4066_9224_428d5972d978.slice/crio-43c7f9560da6bea72544d56f2ea79d613f1cea583438ec4c583f4bda48fc5294 WatchSource:0}: Error finding container 43c7f9560da6bea72544d56f2ea79d613f1cea583438ec4c583f4bda48fc5294: Status 404 returned error can't find the container with id 43c7f9560da6bea72544d56f2ea79d613f1cea583438ec4c583f4bda48fc5294
Sep 09 21:11:18.192237 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:18.192208    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:11:18.192292 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:18.192270    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:11:18.192328 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:18.192302    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:11:18.347068 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:18.347030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" event={"ID":"1b06e898-48d3-4066-9224-428d5972d978","Type":"ContainerStarted","Data":"562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664"}
Sep 09 21:11:18.347288 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:18.347087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" event={"ID":"1b06e898-48d3-4066-9224-428d5972d978","Type":"ContainerStarted","Data":"43c7f9560da6bea72544d56f2ea79d613f1cea583438ec4c583f4bda48fc5294"}
Sep 09 21:11:18.373580 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:18.373527    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" podStartSLOduration=1.373510203 podStartE2EDuration="1.373510203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:18.372052171 +0000 UTC m=+1355.026832330" watchObservedRunningTime="2026-09-09 21:11:18.373510203 +0000 UTC m=+1355.028290362"
Sep 09 21:11:18.898797 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:18.898755    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:19.903779 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:19.903747    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:20.356158 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:20.356122    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:20.357142 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:20.357123    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:21.424047 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.424014    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"]
Sep 09 21:11:21.426519 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.426495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:21.429189 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.429154    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f8b5626b-serving-cert\""
Sep 09 21:11:21.442619 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.442598    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"]
Sep 09 21:11:21.480150 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.480119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-metrics-certs\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:21.480267 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.480209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851f45a1-bcdb-4713-b4aa-81980684aa58-openshift-service-ca\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:21.480267 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.480251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-maas-api-tls\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:21.480342 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.480290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcgcr\" (UniqueName: \"kubernetes.io/projected/851f45a1-bcdb-4713-b4aa-81980684aa58-kube-api-access-dcgcr\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:21.581771 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.581733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-metrics-certs\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:21.581957 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.581785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851f45a1-bcdb-4713-b4aa-81980684aa58-openshift-service-ca\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:21.581957 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.581813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-maas-api-tls\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:21.581957 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.581844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcgcr\" (UniqueName: \"kubernetes.io/projected/851f45a1-bcdb-4713-b4aa-81980684aa58-kube-api-access-dcgcr\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:21.581957 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:11:21.581945    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-serving-cert: secret "maas-api-e2e-mt-f8b5626b-serving-cert" not found
Sep 09 21:11:21.582188 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:11:21.582021    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-maas-api-tls podName:851f45a1-bcdb-4713-b4aa-81980684aa58 nodeName:}" failed. No retries permitted until 2026-09-09 21:11:22.082000075 +0000 UTC m=+1358.736780222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-maas-api-tls") pod "maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz" (UID: "851f45a1-bcdb-4713-b4aa-81980684aa58") : secret "maas-api-e2e-mt-f8b5626b-serving-cert" not found
Sep 09 21:11:21.582522 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.582502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851f45a1-bcdb-4713-b4aa-81980684aa58-openshift-service-ca\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:21.584125 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.584109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-metrics-certs\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:21.599425 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:21.599397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcgcr\" (UniqueName: \"kubernetes.io/projected/851f45a1-bcdb-4713-b4aa-81980684aa58-kube-api-access-dcgcr\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:22.087075 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:22.087044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-maas-api-tls\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:22.089292 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:22.089273    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-maas-api-tls\") pod \"maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:22.337361 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:22.337259    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:22.607053 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:22.607030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"]
Sep 09 21:11:22.609499 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:11:22.609468    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851f45a1_bcdb_4713_b4aa_81980684aa58.slice/crio-032a16259b75ca4b7fa3e7570a8ce2dd2cb9c67a2de107422cd9c5e6de3fbbf1 WatchSource:0}: Error finding container 032a16259b75ca4b7fa3e7570a8ce2dd2cb9c67a2de107422cd9c5e6de3fbbf1: Status 404 returned error can't find the container with id 032a16259b75ca4b7fa3e7570a8ce2dd2cb9c67a2de107422cd9c5e6de3fbbf1
Sep 09 21:11:22.610877 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:22.610856    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:11:23.373867 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:23.373771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz" event={"ID":"851f45a1-bcdb-4713-b4aa-81980684aa58","Type":"ContainerStarted","Data":"6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506"}
Sep 09 21:11:23.374054 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:23.373868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz" event={"ID":"851f45a1-bcdb-4713-b4aa-81980684aa58","Type":"ContainerStarted","Data":"032a16259b75ca4b7fa3e7570a8ce2dd2cb9c67a2de107422cd9c5e6de3fbbf1"}
Sep 09 21:11:23.374054 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:23.373896    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:23.390207 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:23.390130    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz" podStartSLOduration=1.888834504 podStartE2EDuration="2.390110988s" totalImagesPullingTime="501.276484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:23.389679803 +0000 UTC m=+1360.044459962" watchObservedRunningTime="2026-09-09 21:11:23.390110988 +0000 UTC m=+1360.044891149"
Sep 09 21:11:29.385293 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:29.385263    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:11:30.406791 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:30.406757    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:11:40.467538 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:40.467449    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b06e898-48d3-4066-9224-428d5972d978" containerID="562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664" exitCode=0
Sep 09 21:11:40.467538 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:40.467523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" event={"ID":"1b06e898-48d3-4066-9224-428d5972d978","Type":"ContainerDied","Data":"562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664"}
Sep 09 21:11:40.468097 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:40.468079    2578 scope.go:117] "RemoveContainer" containerID="562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664"
Sep 09 21:11:40.469213 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:40.469183    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:11:40.469325 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:40.469263    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:11:40.469325 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:40.469311    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:11:41.474845 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:41.474812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" event={"ID":"1b06e898-48d3-4066-9224-428d5972d978","Type":"ContainerStarted","Data":"4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0"}
Sep 09 21:11:41.899114 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:41.899071    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:41.899330 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:41.899305    2578 patch_prober.go:28] interesting pod/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 21:11:41.899385 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:41.899369    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 21:11:42.898467 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:42.898432    2578 patch_prober.go:28] interesting pod/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 21:11:42.898845 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:42.898489    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 21:11:43.880484 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:43.880458    2578 scope.go:117] "RemoveContainer" containerID="1bc6d0fb7432a656212ffdb031961baa90565f6bc8403ecaacd379fcffa18bf8"
Sep 09 21:11:43.899049 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:43.899025    2578 patch_prober.go:28] interesting pod/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 21:11:43.899417 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:43.899079    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 21:11:44.899048 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:44.899013    2578 patch_prober.go:28] interesting pod/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 21:11:44.899445 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:44.899070    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 21:11:45.899496 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:45.899451    2578 patch_prober.go:28] interesting pod/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 21:11:45.899896 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:45.899523    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 21:11:46.898502 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:46.898461    2578 patch_prober.go:28] interesting pod/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 21:11:46.898694 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:46.898522    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 21:11:47.902994 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:47.902964    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:47.903391 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:47.903327    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:47.903924 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:47.903905    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:11:54.003934 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:11:54.003901    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:12:17.311326 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:17.311286    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:12:29.968902 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:29.968872    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"]
Sep 09 21:12:29.969373 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:29.969065    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz" podUID="851f45a1-bcdb-4713-b4aa-81980684aa58" containerName="maas-api" containerID="cri-o://6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506" gracePeriod=30
Sep 09 21:12:30.226304 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.226242    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:12:30.288114 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.288078    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-maas-api-tls\") pod \"851f45a1-bcdb-4713-b4aa-81980684aa58\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") "
Sep 09 21:12:30.288389 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.288166    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851f45a1-bcdb-4713-b4aa-81980684aa58-openshift-service-ca\") pod \"851f45a1-bcdb-4713-b4aa-81980684aa58\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") "
Sep 09 21:12:30.288389 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.288224    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcgcr\" (UniqueName: \"kubernetes.io/projected/851f45a1-bcdb-4713-b4aa-81980684aa58-kube-api-access-dcgcr\") pod \"851f45a1-bcdb-4713-b4aa-81980684aa58\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") "
Sep 09 21:12:30.288389 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.288313    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-metrics-certs\") pod \"851f45a1-bcdb-4713-b4aa-81980684aa58\" (UID: \"851f45a1-bcdb-4713-b4aa-81980684aa58\") "
Sep 09 21:12:30.289375 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.289309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851f45a1-bcdb-4713-b4aa-81980684aa58-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "851f45a1-bcdb-4713-b4aa-81980684aa58" (UID: "851f45a1-bcdb-4713-b4aa-81980684aa58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:12:30.292039 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.292002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "851f45a1-bcdb-4713-b4aa-81980684aa58" (UID: "851f45a1-bcdb-4713-b4aa-81980684aa58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:30.292039 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.292022    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "851f45a1-bcdb-4713-b4aa-81980684aa58" (UID: "851f45a1-bcdb-4713-b4aa-81980684aa58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:30.292795 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.292768    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851f45a1-bcdb-4713-b4aa-81980684aa58-kube-api-access-dcgcr" (OuterVolumeSpecName: "kube-api-access-dcgcr") pod "851f45a1-bcdb-4713-b4aa-81980684aa58" (UID: "851f45a1-bcdb-4713-b4aa-81980684aa58"). InnerVolumeSpecName "kube-api-access-dcgcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:30.389135 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.389100    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:12:30.389135 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.389129    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851f45a1-bcdb-4713-b4aa-81980684aa58-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:12:30.389135 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.389138    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851f45a1-bcdb-4713-b4aa-81980684aa58-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:12:30.389382 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.389147    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcgcr\" (UniqueName: \"kubernetes.io/projected/851f45a1-bcdb-4713-b4aa-81980684aa58-kube-api-access-dcgcr\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:12:30.701544 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.701506    2578 generic.go:356] "Generic (PLEG): container finished" podID="851f45a1-bcdb-4713-b4aa-81980684aa58" containerID="6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506" exitCode=0
Sep 09 21:12:30.701730 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.701556    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"
Sep 09 21:12:30.701730 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.701568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz" event={"ID":"851f45a1-bcdb-4713-b4aa-81980684aa58","Type":"ContainerDied","Data":"6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506"}
Sep 09 21:12:30.701730 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.701621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz" event={"ID":"851f45a1-bcdb-4713-b4aa-81980684aa58","Type":"ContainerDied","Data":"032a16259b75ca4b7fa3e7570a8ce2dd2cb9c67a2de107422cd9c5e6de3fbbf1"}
Sep 09 21:12:30.701730 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.701650    2578 scope.go:117] "RemoveContainer" containerID="6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506"
Sep 09 21:12:30.711014 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.710993    2578 scope.go:117] "RemoveContainer" containerID="6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506"
Sep 09 21:12:30.711328 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:12:30.711310    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506\": container with ID starting with 6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506 not found: ID does not exist" containerID="6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506"
Sep 09 21:12:30.711397 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.711340    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506"} err="failed to get container status \"6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506\": rpc error: code = NotFound desc = could not find container \"6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506\": container with ID starting with 6398188befecf0b05f25d1a40b464cb0979700def54e0c3ba27a6afc432da506 not found: ID does not exist"
Sep 09 21:12:30.724112 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.724085    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"]
Sep 09 21:12:30.728990 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:30.728970    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8b5626b-876f4879f-4x6gz"]
Sep 09 21:12:31.841489 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:31.841462    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851f45a1-bcdb-4713-b4aa-81980684aa58" path="/var/lib/kubelet/pods/851f45a1-bcdb-4713-b4aa-81980684aa58/volumes"
Sep 09 21:12:35.906765 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:35.906686    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:12:43.898942 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:43.898913    2578 scope.go:117] "RemoveContainer" containerID="940d6d1ce359242516366cc4093c9aaf816215b180719056923a16ffd681348b"
Sep 09 21:12:43.906704 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:43.906686    2578 scope.go:117] "RemoveContainer" containerID="aab8ab60301aef256bf47d16ce1a92bec82d986e76650b112b1cd6d5d9dba887"
Sep 09 21:12:49.612916 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:12:49.612878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:13:05.702956 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:05.702922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:13:19.116777 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:19.116741    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:13:21.958144 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:21.958106    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"]
Sep 09 21:13:21.958616 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:21.958380    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy" containerID="cri-o://4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0" gracePeriod=30
Sep 09 21:13:27.221266 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.221200    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:13:27.385767 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.385733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-envoy\") pod \"1b06e898-48d3-4066-9224-428d5972d978\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") "
Sep 09 21:13:27.385950 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.385791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-certs\") pod \"1b06e898-48d3-4066-9224-428d5972d978\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") "
Sep 09 21:13:27.385950 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.385814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm9x4\" (UniqueName: \"kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-kube-api-access-rm9x4\") pod \"1b06e898-48d3-4066-9224-428d5972d978\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") "
Sep 09 21:13:27.385950 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.385849    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b06e898-48d3-4066-9224-428d5972d978-istiod-ca-cert\") pod \"1b06e898-48d3-4066-9224-428d5972d978\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") "
Sep 09 21:13:27.385950 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.385874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-credential-socket\") pod \"1b06e898-48d3-4066-9224-428d5972d978\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") "
Sep 09 21:13:27.385950 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.385916    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-socket\") pod \"1b06e898-48d3-4066-9224-428d5972d978\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") "
Sep 09 21:13:27.386257 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.385961    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-istio-token\") pod \"1b06e898-48d3-4066-9224-428d5972d978\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") "
Sep 09 21:13:27.386257 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.385994    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-data\") pod \"1b06e898-48d3-4066-9224-428d5972d978\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") "
Sep 09 21:13:27.386257 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.386018    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b06e898-48d3-4066-9224-428d5972d978-istio-podinfo\") pod \"1b06e898-48d3-4066-9224-428d5972d978\" (UID: \"1b06e898-48d3-4066-9224-428d5972d978\") "
Sep 09 21:13:27.386257 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.386093    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b06e898-48d3-4066-9224-428d5972d978" (UID: "1b06e898-48d3-4066-9224-428d5972d978"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:27.386460 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.386260    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b06e898-48d3-4066-9224-428d5972d978" (UID: "1b06e898-48d3-4066-9224-428d5972d978"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:27.386460 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.386352    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b06e898-48d3-4066-9224-428d5972d978" (UID: "1b06e898-48d3-4066-9224-428d5972d978"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:27.386460 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.386408    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b06e898-48d3-4066-9224-428d5972d978-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b06e898-48d3-4066-9224-428d5972d978" (UID: "1b06e898-48d3-4066-9224-428d5972d978"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:13:27.386460 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.386432    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:13:27.386460 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.386452    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:13:27.386633 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.386598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b06e898-48d3-4066-9224-428d5972d978" (UID: "1b06e898-48d3-4066-9224-428d5972d978"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:27.388125 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.388101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-kube-api-access-rm9x4" (OuterVolumeSpecName: "kube-api-access-rm9x4") pod "1b06e898-48d3-4066-9224-428d5972d978" (UID: "1b06e898-48d3-4066-9224-428d5972d978"). InnerVolumeSpecName "kube-api-access-rm9x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:27.388297 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.388152    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b06e898-48d3-4066-9224-428d5972d978-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b06e898-48d3-4066-9224-428d5972d978" (UID: "1b06e898-48d3-4066-9224-428d5972d978"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:13:27.388542 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.388522    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b06e898-48d3-4066-9224-428d5972d978" (UID: "1b06e898-48d3-4066-9224-428d5972d978"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:27.388610 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.388525    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b06e898-48d3-4066-9224-428d5972d978" (UID: "1b06e898-48d3-4066-9224-428d5972d978"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:27.487216 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.487096    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:13:27.487216 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.487138    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b06e898-48d3-4066-9224-428d5972d978-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:13:27.487216 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.487148    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:13:27.487216 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.487157    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm9x4\" (UniqueName: \"kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-kube-api-access-rm9x4\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:13:27.487216 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.487166    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b06e898-48d3-4066-9224-428d5972d978-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:13:27.487216 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.487200    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06e898-48d3-4066-9224-428d5972d978-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:13:27.487216 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.487209    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b06e898-48d3-4066-9224-428d5972d978-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:13:27.974085 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.974053    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b06e898-48d3-4066-9224-428d5972d978" containerID="4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0" exitCode=0
Sep 09 21:13:27.974268 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.974101    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"
Sep 09 21:13:27.974268 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.974115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" event={"ID":"1b06e898-48d3-4066-9224-428d5972d978","Type":"ContainerDied","Data":"4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0"}
Sep 09 21:13:27.974268 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.974157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr" event={"ID":"1b06e898-48d3-4066-9224-428d5972d978","Type":"ContainerDied","Data":"43c7f9560da6bea72544d56f2ea79d613f1cea583438ec4c583f4bda48fc5294"}
Sep 09 21:13:27.974268 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.974207    2578 scope.go:117] "RemoveContainer" containerID="4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0"
Sep 09 21:13:27.982451 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.982431    2578 scope.go:117] "RemoveContainer" containerID="562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664"
Sep 09 21:13:27.990041 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.990020    2578 scope.go:117] "RemoveContainer" containerID="4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0"
Sep 09 21:13:27.990358 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:13:27.990337    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0\": container with ID starting with 4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0 not found: ID does not exist" containerID="4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0"
Sep 09 21:13:27.990410 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.990368    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0"} err="failed to get container status \"4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0\": rpc error: code = NotFound desc = could not find container \"4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0\": container with ID starting with 4c6493ef151327dec79b6dd728acdd024b3d112b9f370d4f0fe69da3ab8ee4c0 not found: ID does not exist"
Sep 09 21:13:27.990410 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.990385    2578 scope.go:117] "RemoveContainer" containerID="562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664"
Sep 09 21:13:27.990647 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:13:27.990630    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664\": container with ID starting with 562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664 not found: ID does not exist" containerID="562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664"
Sep 09 21:13:27.990689 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.990653    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664"} err="failed to get container status \"562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664\": rpc error: code = NotFound desc = could not find container \"562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664\": container with ID starting with 562402a5bb7260ae5517b71b8c43b8cf3bf8147d5ca38bc4f4b94e5b2c453664 not found: ID does not exist"
Sep 09 21:13:27.991046 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.991030    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"]
Sep 09 21:13:27.997110 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:27.997087    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f8b5626b-openshift-default-6cd576db96-xwqqr"]
Sep 09 21:13:29.840778 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:29.840744    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b06e898-48d3-4066-9224-428d5972d978" path="/var/lib/kubelet/pods/1b06e898-48d3-4066-9224-428d5972d978/volumes"
Sep 09 21:13:39.606851 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:39.606817    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:13:41.713969 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.713935    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"]
Sep 09 21:13:41.714339 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.714256    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy"
Sep 09 21:13:41.714339 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.714268    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy"
Sep 09 21:13:41.714339 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.714276    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851f45a1-bcdb-4713-b4aa-81980684aa58" containerName="maas-api"
Sep 09 21:13:41.714339 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.714282    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="851f45a1-bcdb-4713-b4aa-81980684aa58" containerName="maas-api"
Sep 09 21:13:41.714474 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.714356    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy"
Sep 09 21:13:41.714474 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.714365    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="851f45a1-bcdb-4713-b4aa-81980684aa58" containerName="maas-api"
Sep 09 21:13:41.714474 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.714372    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy"
Sep 09 21:13:41.717599 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.717581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.720499 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.720473    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3d3cac\"/\"default-dockercfg-vb4x7\""
Sep 09 21:13:41.720627 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.720473    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3d3cac\"/\"openshift-service-ca.crt\""
Sep 09 21:13:41.720627 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.720478    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3d3cac\"/\"sub-test-model-3d3cac-kserve-self-signed-certs\""
Sep 09 21:13:41.720627 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.720556    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3d3cac\"/\"kube-root-ca.crt\""
Sep 09 21:13:41.730401 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.730378    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"]
Sep 09 21:13:41.819987 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.819954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dp6r\" (UniqueName: \"kubernetes.io/projected/805fb371-b94c-4baa-8e73-e6a937be982d-kube-api-access-2dp6r\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.820138 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.819993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805fb371-b94c-4baa-8e73-e6a937be982d-tls-certs\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.820138 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.820058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-model-cache\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.820138 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.820076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-tmp-dir\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.820138 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.820136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-home\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.820319 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.820154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-dshm\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.921220 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.921187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-dshm\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.921430 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.921232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dp6r\" (UniqueName: \"kubernetes.io/projected/805fb371-b94c-4baa-8e73-e6a937be982d-kube-api-access-2dp6r\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.921430 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.921258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805fb371-b94c-4baa-8e73-e6a937be982d-tls-certs\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.921430 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.921302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-model-cache\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.921430 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.921328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-tmp-dir\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.921430 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.921383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-home\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.921762 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.921679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-home\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.921762 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.921683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-model-cache\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.921762 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.921719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-tmp-dir\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.923446 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.923425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-dshm\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.923754 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.923736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805fb371-b94c-4baa-8e73-e6a937be982d-tls-certs\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:41.928789 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:41.928770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dp6r\" (UniqueName: \"kubernetes.io/projected/805fb371-b94c-4baa-8e73-e6a937be982d-kube-api-access-2dp6r\") pod \"sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:42.026785 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:42.026697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:42.295648 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:42.295621    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"]
Sep 09 21:13:42.298273 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:13:42.298245    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod805fb371_b94c_4baa_8e73_e6a937be982d.slice/crio-5f4df1d175952f4a73010f412abcc7ae3d512f242828110cbffdea8d38b8d528 WatchSource:0}: Error finding container 5f4df1d175952f4a73010f412abcc7ae3d512f242828110cbffdea8d38b8d528: Status 404 returned error can't find the container with id 5f4df1d175952f4a73010f412abcc7ae3d512f242828110cbffdea8d38b8d528
Sep 09 21:13:43.046589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:43.046552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf" event={"ID":"805fb371-b94c-4baa-8e73-e6a937be982d","Type":"ContainerStarted","Data":"1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8"}
Sep 09 21:13:43.046999 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:43.046597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf" event={"ID":"805fb371-b94c-4baa-8e73-e6a937be982d","Type":"ContainerStarted","Data":"5f4df1d175952f4a73010f412abcc7ae3d512f242828110cbffdea8d38b8d528"}
Sep 09 21:13:43.063411 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:43.063366    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf" podStartSLOduration=2.0633513 podStartE2EDuration="2.0633513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:43.061791526 +0000 UTC m=+1499.716571684" watchObservedRunningTime="2026-09-09 21:13:43.0633513 +0000 UTC m=+1499.718131483"
Sep 09 21:13:43.930271 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:43.930245    2578 scope.go:117] "RemoveContainer" containerID="f17c3b56670bbb47c12b43407792b702ecbd07274ef6a6cfa21787e2ee22f40e"
Sep 09 21:13:52.027361 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:52.027320    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:52.027841 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:52.027375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:52.040277 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:52.040253    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:52.101970 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:52.101943    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:13:54.804521 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:13:54.804485    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:14:09.904220 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:09.904190    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:14:21.409337 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.409305    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"]
Sep 09 21:14:21.409948 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.409824    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy"
Sep 09 21:14:21.409948 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.409862    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b06e898-48d3-4066-9224-428d5972d978" containerName="istio-proxy"
Sep 09 21:14:21.412951 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.412933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:21.415903 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.415880    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dc3099-serving-cert\""
Sep 09 21:14:21.423819 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.423796    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"]
Sep 09 21:14:21.473438 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.473413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-maas-api-tls\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:21.473553 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.473466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-metrics-certs\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:21.473553 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.473490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee70998b-6224-435f-a268-e3a57f955871-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:21.473627 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.473555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw6xc\" (UniqueName: \"kubernetes.io/projected/ee70998b-6224-435f-a268-e3a57f955871-kube-api-access-kw6xc\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:21.574288 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.574254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-maas-api-tls\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:21.574448 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.574317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-metrics-certs\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:21.574448 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.574336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee70998b-6224-435f-a268-e3a57f955871-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:21.574448 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.574361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw6xc\" (UniqueName: \"kubernetes.io/projected/ee70998b-6224-435f-a268-e3a57f955871-kube-api-access-kw6xc\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:21.574448 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:14:21.574398    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-serving-cert: secret "maas-api-e2e-api-a-dc3099-serving-cert" not found
Sep 09 21:14:21.574588 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:14:21.574460    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-maas-api-tls podName:ee70998b-6224-435f-a268-e3a57f955871 nodeName:}" failed. No retries permitted until 2026-09-09 21:14:22.074442928 +0000 UTC m=+1538.729223064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-maas-api-tls") pod "maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb" (UID: "ee70998b-6224-435f-a268-e3a57f955871") : secret "maas-api-e2e-api-a-dc3099-serving-cert" not found
Sep 09 21:14:21.575104 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.575086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee70998b-6224-435f-a268-e3a57f955871-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:21.576819 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.576793    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-metrics-certs\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:21.586476 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:21.586456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw6xc\" (UniqueName: \"kubernetes.io/projected/ee70998b-6224-435f-a268-e3a57f955871-kube-api-access-kw6xc\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:22.078134 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:22.078105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-maas-api-tls\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:22.080359 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:22.080330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-maas-api-tls\") pod \"maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:22.323986 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:22.323949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:22.595247 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:22.595221    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"]
Sep 09 21:14:22.597650 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:14:22.597623    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee70998b_6224_435f_a268_e3a57f955871.slice/crio-0921144d4f01a2f36d656a27cbe2f2c7c24f340ca339f0651aad1d05a90bdb94 WatchSource:0}: Error finding container 0921144d4f01a2f36d656a27cbe2f2c7c24f340ca339f0651aad1d05a90bdb94: Status 404 returned error can't find the container with id 0921144d4f01a2f36d656a27cbe2f2c7c24f340ca339f0651aad1d05a90bdb94
Sep 09 21:14:23.229207 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:23.229165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb" event={"ID":"ee70998b-6224-435f-a268-e3a57f955871","Type":"ContainerStarted","Data":"a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4"}
Sep 09 21:14:23.229327 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:23.229234    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb" event={"ID":"ee70998b-6224-435f-a268-e3a57f955871","Type":"ContainerStarted","Data":"0921144d4f01a2f36d656a27cbe2f2c7c24f340ca339f0651aad1d05a90bdb94"}
Sep 09 21:14:23.229327 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:23.229288    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:23.247406 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:23.247364    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb" podStartSLOduration=1.696020404 podStartE2EDuration="2.247343674s" totalImagesPullingTime="551.32327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:23.246464875 +0000 UTC m=+1539.901245046" watchObservedRunningTime="2026-09-09 21:14:23.247343674 +0000 UTC m=+1539.902123832"
Sep 09 21:14:29.239439 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:29.239410    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:14:31.311945 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:31.311909    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:14:50.542440 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.542404    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"]
Sep 09 21:14:50.546030 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.546010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:50.548442 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.548418    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bd9741-serving-cert\""
Sep 09 21:14:50.558467 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.558444    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"]
Sep 09 21:14:50.631983 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.631953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-metrics-certs\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:50.632122 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.631988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-maas-api-tls\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:50.632122 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.632013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00628624-0172-46e9-ad60-83dbe4e2228f-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:50.632122 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.632098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9qzx\" (UniqueName: \"kubernetes.io/projected/00628624-0172-46e9-ad60-83dbe4e2228f-kube-api-access-w9qzx\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:50.733515 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.733480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9qzx\" (UniqueName: \"kubernetes.io/projected/00628624-0172-46e9-ad60-83dbe4e2228f-kube-api-access-w9qzx\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:50.733692 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.733558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-metrics-certs\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:50.733692 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.733578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-maas-api-tls\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:50.733692 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.733594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00628624-0172-46e9-ad60-83dbe4e2228f-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:50.733824 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:14:50.733709    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-serving-cert: secret "maas-api-e2e-api-b-bd9741-serving-cert" not found
Sep 09 21:14:50.733824 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:14:50.733787    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-maas-api-tls podName:00628624-0172-46e9-ad60-83dbe4e2228f nodeName:}" failed. No retries permitted until 2026-09-09 21:14:51.233767592 +0000 UTC m=+1567.888547727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-maas-api-tls") pod "maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn" (UID: "00628624-0172-46e9-ad60-83dbe4e2228f") : secret "maas-api-e2e-api-b-bd9741-serving-cert" not found
Sep 09 21:14:50.734197 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.734153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00628624-0172-46e9-ad60-83dbe4e2228f-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:50.736112 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.736096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-metrics-certs\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:50.749973 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:50.749952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9qzx\" (UniqueName: \"kubernetes.io/projected/00628624-0172-46e9-ad60-83dbe4e2228f-kube-api-access-w9qzx\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:51.237924 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:51.237887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-maas-api-tls\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:51.240160 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:51.240139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-maas-api-tls\") pod \"maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:51.456619 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:51.456586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:51.727476 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:51.727449    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"]
Sep 09 21:14:51.729853 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:14:51.729826    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00628624_0172_46e9_ad60_83dbe4e2228f.slice/crio-173ddfa23492fdad98ce4ca412fa61e13d9b0ce65b8343d22d9aa575b43287b0 WatchSource:0}: Error finding container 173ddfa23492fdad98ce4ca412fa61e13d9b0ce65b8343d22d9aa575b43287b0: Status 404 returned error can't find the container with id 173ddfa23492fdad98ce4ca412fa61e13d9b0ce65b8343d22d9aa575b43287b0
Sep 09 21:14:52.364393 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:52.364307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn" event={"ID":"00628624-0172-46e9-ad60-83dbe4e2228f","Type":"ContainerStarted","Data":"e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb"}
Sep 09 21:14:52.364393 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:52.364364    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn" event={"ID":"00628624-0172-46e9-ad60-83dbe4e2228f","Type":"ContainerStarted","Data":"173ddfa23492fdad98ce4ca412fa61e13d9b0ce65b8343d22d9aa575b43287b0"}
Sep 09 21:14:52.364393 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:52.364386    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:14:52.384071 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:52.384023    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn" podStartSLOduration=2.02943054 podStartE2EDuration="2.384008754s" totalImagesPullingTime="354.578214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:52.382322233 +0000 UTC m=+1569.037102392" watchObservedRunningTime="2026-09-09 21:14:52.384008754 +0000 UTC m=+1569.038788911"
Sep 09 21:14:52.690004 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:52.689927    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"]
Sep 09 21:14:56.106089 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:56.106052    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:14:58.374865 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:14:58.374834    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:15:00.172471 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:00.172433    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk"]
Sep 09 21:15:00.176077 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:00.176056    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk"
Sep 09 21:15:00.178814 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:00.178791    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6l2wq\""
Sep 09 21:15:00.186166 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:00.186145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk"
Sep 09 21:15:00.194307 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:00.194281    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk"]
Sep 09 21:15:00.459588 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:00.459564    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk"]
Sep 09 21:15:00.461710 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:15:00.461683    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd48218c_620c_4a04_9b6c_39c942ab5da2.slice/crio-a8fe294024f2edf7dc1bece44457b441f5b1f91138d72ab4d9c658789d5f5991 WatchSource:0}: Error finding container a8fe294024f2edf7dc1bece44457b441f5b1f91138d72ab4d9c658789d5f5991: Status 404 returned error can't find the container with id a8fe294024f2edf7dc1bece44457b441f5b1f91138d72ab4d9c658789d5f5991
Sep 09 21:15:01.409142 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:01.409108    2578 generic.go:356] "Generic (PLEG): container finished" podID="bd48218c-620c-4a04-9b6c-39c942ab5da2" containerID="f24ec610ab1f87b5f079e9b1c9bc927b04e547a407f4e949ba002f8371daee28" exitCode=0
Sep 09 21:15:01.409535 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:01.409194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk" event={"ID":"bd48218c-620c-4a04-9b6c-39c942ab5da2","Type":"ContainerDied","Data":"f24ec610ab1f87b5f079e9b1c9bc927b04e547a407f4e949ba002f8371daee28"}
Sep 09 21:15:01.409535 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:01.409232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk" event={"ID":"bd48218c-620c-4a04-9b6c-39c942ab5da2","Type":"ContainerStarted","Data":"a8fe294024f2edf7dc1bece44457b441f5b1f91138d72ab4d9c658789d5f5991"}
Sep 09 21:15:02.540393 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:02.540368    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk"
Sep 09 21:15:03.420763 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:03.420727    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk" event={"ID":"bd48218c-620c-4a04-9b6c-39c942ab5da2","Type":"ContainerDied","Data":"a8fe294024f2edf7dc1bece44457b441f5b1f91138d72ab4d9c658789d5f5991"}
Sep 09 21:15:03.420763 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:03.420754    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk"
Sep 09 21:15:03.420965 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:03.420785    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8fe294024f2edf7dc1bece44457b441f5b1f91138d72ab4d9c658789d5f5991"
Sep 09 21:15:07.709273 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.709230    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf" podUID="805fb371-b94c-4baa-8e73-e6a937be982d" containerName="main" containerID="cri-o://1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8" gracePeriod=45
Sep 09 21:15:07.970313 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.970257    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:15:07.988798 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.988769    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-home\") pod \"805fb371-b94c-4baa-8e73-e6a937be982d\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") "
Sep 09 21:15:07.988940 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.988856    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805fb371-b94c-4baa-8e73-e6a937be982d-tls-certs\") pod \"805fb371-b94c-4baa-8e73-e6a937be982d\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") "
Sep 09 21:15:07.988940 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.988884    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-tmp-dir\") pod \"805fb371-b94c-4baa-8e73-e6a937be982d\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") "
Sep 09 21:15:07.988940 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.988903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-dshm\") pod \"805fb371-b94c-4baa-8e73-e6a937be982d\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") "
Sep 09 21:15:07.989121 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.988943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-model-cache\") pod \"805fb371-b94c-4baa-8e73-e6a937be982d\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") "
Sep 09 21:15:07.989121 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.988982    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dp6r\" (UniqueName: \"kubernetes.io/projected/805fb371-b94c-4baa-8e73-e6a937be982d-kube-api-access-2dp6r\") pod \"805fb371-b94c-4baa-8e73-e6a937be982d\" (UID: \"805fb371-b94c-4baa-8e73-e6a937be982d\") "
Sep 09 21:15:07.989121 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.989060    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-home" (OuterVolumeSpecName: "home") pod "805fb371-b94c-4baa-8e73-e6a937be982d" (UID: "805fb371-b94c-4baa-8e73-e6a937be982d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:07.989299 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.989287    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-home\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:07.989424 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.989392    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "805fb371-b94c-4baa-8e73-e6a937be982d" (UID: "805fb371-b94c-4baa-8e73-e6a937be982d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:07.989556 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.989472    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-model-cache" (OuterVolumeSpecName: "model-cache") pod "805fb371-b94c-4baa-8e73-e6a937be982d" (UID: "805fb371-b94c-4baa-8e73-e6a937be982d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:07.991301 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.991277    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805fb371-b94c-4baa-8e73-e6a937be982d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "805fb371-b94c-4baa-8e73-e6a937be982d" (UID: "805fb371-b94c-4baa-8e73-e6a937be982d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:07.991301 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.991287    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805fb371-b94c-4baa-8e73-e6a937be982d-kube-api-access-2dp6r" (OuterVolumeSpecName: "kube-api-access-2dp6r") pod "805fb371-b94c-4baa-8e73-e6a937be982d" (UID: "805fb371-b94c-4baa-8e73-e6a937be982d"). InnerVolumeSpecName "kube-api-access-2dp6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:07.991491 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:07.991476    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-dshm" (OuterVolumeSpecName: "dshm") pod "805fb371-b94c-4baa-8e73-e6a937be982d" (UID: "805fb371-b94c-4baa-8e73-e6a937be982d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:08.090135 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.090098    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-model-cache\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:08.090135 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.090128    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dp6r\" (UniqueName: \"kubernetes.io/projected/805fb371-b94c-4baa-8e73-e6a937be982d-kube-api-access-2dp6r\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:08.090135 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.090139    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805fb371-b94c-4baa-8e73-e6a937be982d-tls-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:08.090397 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.090149    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-tmp-dir\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:08.090397 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.090157    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805fb371-b94c-4baa-8e73-e6a937be982d-dshm\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:08.444595 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.444561    2578 generic.go:356] "Generic (PLEG): container finished" podID="805fb371-b94c-4baa-8e73-e6a937be982d" containerID="1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8" exitCode=0
Sep 09 21:15:08.444748 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.444603    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"
Sep 09 21:15:08.444748 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.444640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf" event={"ID":"805fb371-b94c-4baa-8e73-e6a937be982d","Type":"ContainerDied","Data":"1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8"}
Sep 09 21:15:08.444748 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.444681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf" event={"ID":"805fb371-b94c-4baa-8e73-e6a937be982d","Type":"ContainerDied","Data":"5f4df1d175952f4a73010f412abcc7ae3d512f242828110cbffdea8d38b8d528"}
Sep 09 21:15:08.444748 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.444699    2578 scope.go:117] "RemoveContainer" containerID="1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8"
Sep 09 21:15:08.453211 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.453196    2578 scope.go:117] "RemoveContainer" containerID="1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8"
Sep 09 21:15:08.453461 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:15:08.453442    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8\": container with ID starting with 1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8 not found: ID does not exist" containerID="1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8"
Sep 09 21:15:08.453510 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.453469    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8"} err="failed to get container status \"1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8\": rpc error: code = NotFound desc = could not find container \"1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8\": container with ID starting with 1004121806a02f5c27df962a997d54c564466e2de18d4c183a8a8658758fe1f8 not found: ID does not exist"
Sep 09 21:15:08.464473 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.464450    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"]
Sep 09 21:15:08.467981 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:08.467962    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/sub-test-model-3d3cac-kserve-67fcd9fccc-k45rf"]
Sep 09 21:15:09.840287 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:09.840260    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="805fb371-b94c-4baa-8e73-e6a937be982d" path="/var/lib/kubelet/pods/805fb371-b94c-4baa-8e73-e6a937be982d/volumes"
Sep 09 21:15:21.498077 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:21.498042    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:15:26.008492 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.008455    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"]
Sep 09 21:15:26.008875 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.008798    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd48218c-620c-4a04-9b6c-39c942ab5da2" containerName="cleanup"
Sep 09 21:15:26.008875 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.008810    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd48218c-620c-4a04-9b6c-39c942ab5da2" containerName="cleanup"
Sep 09 21:15:26.008875 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.008828    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805fb371-b94c-4baa-8e73-e6a937be982d" containerName="main"
Sep 09 21:15:26.008875 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.008834    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="805fb371-b94c-4baa-8e73-e6a937be982d" containerName="main"
Sep 09 21:15:26.009040 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.008885    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="805fb371-b94c-4baa-8e73-e6a937be982d" containerName="main"
Sep 09 21:15:26.009040 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.008896    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd48218c-620c-4a04-9b6c-39c942ab5da2" containerName="cleanup"
Sep 09 21:15:26.012189 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.012159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.015308 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.015281    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8db5a7\"/\"openshift-service-ca.crt\""
Sep 09 21:15:26.015441 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.015322    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8db5a7\"/\"rate-test-model-8db5a7-kserve-self-signed-certs\""
Sep 09 21:15:26.015441 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.015353    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8db5a7\"/\"default-dockercfg-bf86r\""
Sep 09 21:15:26.015441 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.015370    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8db5a7\"/\"kube-root-ca.crt\""
Sep 09 21:15:26.022939 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.022915    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"]
Sep 09 21:15:26.154373 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.154333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9031446-060d-406f-94ba-c5f741728b5a-tls-certs\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.154373 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.154373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-tmp-dir\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.154620 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.154400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-model-cache\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.154620 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.154471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcjg5\" (UniqueName: \"kubernetes.io/projected/f9031446-060d-406f-94ba-c5f741728b5a-kube-api-access-jcjg5\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.154620 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.154497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-dshm\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.154620 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.154585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-home\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.255061 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.255023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-home\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.255316 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.255073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9031446-060d-406f-94ba-c5f741728b5a-tls-certs\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.255316 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.255090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-tmp-dir\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.255316 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.255107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-model-cache\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.255316 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.255143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcjg5\" (UniqueName: \"kubernetes.io/projected/f9031446-060d-406f-94ba-c5f741728b5a-kube-api-access-jcjg5\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.255316 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.255163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-dshm\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.255599 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.255504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-tmp-dir\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.255599 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.255534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-home\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.255670 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.255598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-model-cache\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.257425 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.257403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-dshm\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.257703 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.257684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9031446-060d-406f-94ba-c5f741728b5a-tls-certs\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.263308 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.263253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcjg5\" (UniqueName: \"kubernetes.io/projected/f9031446-060d-406f-94ba-c5f741728b5a-kube-api-access-jcjg5\") pod \"rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.321986 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.321957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:26.590546 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:26.590518    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"]
Sep 09 21:15:26.593006 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:15:26.592974    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9031446_060d_406f_94ba_c5f741728b5a.slice/crio-41be8cb242a9bef0c2fdc7f194e6ab3ca12b7d05b79c3b411e99c6c4d57a76d8 WatchSource:0}: Error finding container 41be8cb242a9bef0c2fdc7f194e6ab3ca12b7d05b79c3b411e99c6c4d57a76d8: Status 404 returned error can't find the container with id 41be8cb242a9bef0c2fdc7f194e6ab3ca12b7d05b79c3b411e99c6c4d57a76d8
Sep 09 21:15:27.534799 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:27.534760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv" event={"ID":"f9031446-060d-406f-94ba-c5f741728b5a","Type":"ContainerStarted","Data":"f3fb752817125a933eed54d822ad82c7979edfab1fa46523f557920e21851f3d"}
Sep 09 21:15:27.535220 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:27.534811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv" event={"ID":"f9031446-060d-406f-94ba-c5f741728b5a","Type":"ContainerStarted","Data":"41be8cb242a9bef0c2fdc7f194e6ab3ca12b7d05b79c3b411e99c6c4d57a76d8"}
Sep 09 21:15:27.553009 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:27.552947    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv" podStartSLOduration=2.5529280930000002 podStartE2EDuration="2.552928093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:15:27.551659198 +0000 UTC m=+1604.206439356" watchObservedRunningTime="2026-09-09 21:15:27.552928093 +0000 UTC m=+1604.207708252"
Sep 09 21:15:32.701522 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:32.701488    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"]
Sep 09 21:15:32.702097 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:32.701693    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb" podUID="ee70998b-6224-435f-a268-e3a57f955871" containerName="maas-api" containerID="cri-o://a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4" gracePeriod=30
Sep 09 21:15:32.941230 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:32.941200    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:15:33.021858 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.021793    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-metrics-certs\") pod \"ee70998b-6224-435f-a268-e3a57f955871\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") "
Sep 09 21:15:33.021858 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.021841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee70998b-6224-435f-a268-e3a57f955871-openshift-service-ca\") pod \"ee70998b-6224-435f-a268-e3a57f955871\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") "
Sep 09 21:15:33.022080 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.021866    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw6xc\" (UniqueName: \"kubernetes.io/projected/ee70998b-6224-435f-a268-e3a57f955871-kube-api-access-kw6xc\") pod \"ee70998b-6224-435f-a268-e3a57f955871\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") "
Sep 09 21:15:33.022080 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.021911    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-maas-api-tls\") pod \"ee70998b-6224-435f-a268-e3a57f955871\" (UID: \"ee70998b-6224-435f-a268-e3a57f955871\") "
Sep 09 21:15:33.022287 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.022258    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee70998b-6224-435f-a268-e3a57f955871-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee70998b-6224-435f-a268-e3a57f955871" (UID: "ee70998b-6224-435f-a268-e3a57f955871"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:15:33.023963 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.023926    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ee70998b-6224-435f-a268-e3a57f955871" (UID: "ee70998b-6224-435f-a268-e3a57f955871"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:33.024065 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.024007    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee70998b-6224-435f-a268-e3a57f955871" (UID: "ee70998b-6224-435f-a268-e3a57f955871"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:33.024065 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.024007    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee70998b-6224-435f-a268-e3a57f955871-kube-api-access-kw6xc" (OuterVolumeSpecName: "kube-api-access-kw6xc") pod "ee70998b-6224-435f-a268-e3a57f955871" (UID: "ee70998b-6224-435f-a268-e3a57f955871"). InnerVolumeSpecName "kube-api-access-kw6xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:33.122584 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.122553    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:33.122584 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.122579    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee70998b-6224-435f-a268-e3a57f955871-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:33.122584 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.122589    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw6xc\" (UniqueName: \"kubernetes.io/projected/ee70998b-6224-435f-a268-e3a57f955871-kube-api-access-kw6xc\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:33.122827 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.122598    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee70998b-6224-435f-a268-e3a57f955871-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:33.561461 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.561425    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee70998b-6224-435f-a268-e3a57f955871" containerID="a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4" exitCode=0
Sep 09 21:15:33.561630 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.561469    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"
Sep 09 21:15:33.561630 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.561486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb" event={"ID":"ee70998b-6224-435f-a268-e3a57f955871","Type":"ContainerDied","Data":"a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4"}
Sep 09 21:15:33.561630 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.561533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb" event={"ID":"ee70998b-6224-435f-a268-e3a57f955871","Type":"ContainerDied","Data":"0921144d4f01a2f36d656a27cbe2f2c7c24f340ca339f0651aad1d05a90bdb94"}
Sep 09 21:15:33.561630 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.561556    2578 scope.go:117] "RemoveContainer" containerID="a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4"
Sep 09 21:15:33.570992 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.570973    2578 scope.go:117] "RemoveContainer" containerID="a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4"
Sep 09 21:15:33.571289 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:15:33.571270    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4\": container with ID starting with a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4 not found: ID does not exist" containerID="a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4"
Sep 09 21:15:33.571368 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.571298    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4"} err="failed to get container status \"a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4\": rpc error: code = NotFound desc = could not find container \"a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4\": container with ID starting with a0e26df8be5b39319063c7047b22acb3e242e806e693cd365bdb66bc8fb123b4 not found: ID does not exist"
Sep 09 21:15:33.583789 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.583768    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"]
Sep 09 21:15:33.589567 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.589547    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc3099-6455c9558f-nmlzb"]
Sep 09 21:15:33.846259 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:33.846218    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee70998b-6224-435f-a268-e3a57f955871" path="/var/lib/kubelet/pods/ee70998b-6224-435f-a268-e3a57f955871/volumes"
Sep 09 21:15:34.815018 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:34.814980    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:15:36.322771 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:36.322693    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:36.322771 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:36.322739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:36.335528 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:36.335499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:36.587651 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:36.587625    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:15:43.957902 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:43.957873    2578 scope.go:117] "RemoveContainer" containerID="8007458adef6e7ee36a2b46921a98d076fab60bd4d3b27c1673ec1eebea6a4c8"
Sep 09 21:15:53.906638 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:53.906605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:15:55.593935 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.593903    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"]
Sep 09 21:15:55.594331 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.594113    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn" podUID="00628624-0172-46e9-ad60-83dbe4e2228f" containerName="maas-api" containerID="cri-o://e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb" gracePeriod=30
Sep 09 21:15:55.621766 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.621741    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk"]
Sep 09 21:15:55.625340 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.625312    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bd9741-29816475-ssxqk"]
Sep 09 21:15:55.841526 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.841486    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:15:55.847266 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.845362    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd48218c-620c-4a04-9b6c-39c942ab5da2" path="/var/lib/kubelet/pods/bd48218c-620c-4a04-9b6c-39c942ab5da2/volumes"
Sep 09 21:15:55.915281 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.915253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-maas-api-tls\") pod \"00628624-0172-46e9-ad60-83dbe4e2228f\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") "
Sep 09 21:15:55.915455 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.915320    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9qzx\" (UniqueName: \"kubernetes.io/projected/00628624-0172-46e9-ad60-83dbe4e2228f-kube-api-access-w9qzx\") pod \"00628624-0172-46e9-ad60-83dbe4e2228f\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") "
Sep 09 21:15:55.915455 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.915357    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-metrics-certs\") pod \"00628624-0172-46e9-ad60-83dbe4e2228f\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") "
Sep 09 21:15:55.915455 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.915377    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00628624-0172-46e9-ad60-83dbe4e2228f-openshift-service-ca\") pod \"00628624-0172-46e9-ad60-83dbe4e2228f\" (UID: \"00628624-0172-46e9-ad60-83dbe4e2228f\") "
Sep 09 21:15:55.915905 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.915869    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00628624-0172-46e9-ad60-83dbe4e2228f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00628624-0172-46e9-ad60-83dbe4e2228f" (UID: "00628624-0172-46e9-ad60-83dbe4e2228f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:15:55.917506 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.917478    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "00628624-0172-46e9-ad60-83dbe4e2228f" (UID: "00628624-0172-46e9-ad60-83dbe4e2228f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:55.917592 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.917520    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00628624-0172-46e9-ad60-83dbe4e2228f-kube-api-access-w9qzx" (OuterVolumeSpecName: "kube-api-access-w9qzx") pod "00628624-0172-46e9-ad60-83dbe4e2228f" (UID: "00628624-0172-46e9-ad60-83dbe4e2228f"). InnerVolumeSpecName "kube-api-access-w9qzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:55.917592 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:55.917521    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00628624-0172-46e9-ad60-83dbe4e2228f" (UID: "00628624-0172-46e9-ad60-83dbe4e2228f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:56.016102 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.016070    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:56.016102 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.016097    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9qzx\" (UniqueName: \"kubernetes.io/projected/00628624-0172-46e9-ad60-83dbe4e2228f-kube-api-access-w9qzx\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:56.016102 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.016108    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00628624-0172-46e9-ad60-83dbe4e2228f-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:56.016389 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.016117    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00628624-0172-46e9-ad60-83dbe4e2228f-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:15:56.667727 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.667696    2578 generic.go:356] "Generic (PLEG): container finished" podID="00628624-0172-46e9-ad60-83dbe4e2228f" containerID="e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb" exitCode=0
Sep 09 21:15:56.668115 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.667756    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"
Sep 09 21:15:56.668115 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.667783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn" event={"ID":"00628624-0172-46e9-ad60-83dbe4e2228f","Type":"ContainerDied","Data":"e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb"}
Sep 09 21:15:56.668115 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.667836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn" event={"ID":"00628624-0172-46e9-ad60-83dbe4e2228f","Type":"ContainerDied","Data":"173ddfa23492fdad98ce4ca412fa61e13d9b0ce65b8343d22d9aa575b43287b0"}
Sep 09 21:15:56.668115 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.667859    2578 scope.go:117] "RemoveContainer" containerID="e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb"
Sep 09 21:15:56.677313 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.677292    2578 scope.go:117] "RemoveContainer" containerID="e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb"
Sep 09 21:15:56.677564 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:15:56.677545    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb\": container with ID starting with e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb not found: ID does not exist" containerID="e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb"
Sep 09 21:15:56.677617 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.677571    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb"} err="failed to get container status \"e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb\": rpc error: code = NotFound desc = could not find container \"e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb\": container with ID starting with e4177c96b295c08f1a5df3ef2f54b340998adb356a4af4cf77712361286420eb not found: ID does not exist"
Sep 09 21:15:56.688877 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.688854    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"]
Sep 09 21:15:56.691249 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:56.691227    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd9741-74b8d966c9-84hxn"]
Sep 09 21:15:57.841069 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:15:57.841035    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00628624-0172-46e9-ad60-83dbe4e2228f" path="/var/lib/kubelet/pods/00628624-0172-46e9-ad60-83dbe4e2228f/volumes"
Sep 09 21:16:09.702494 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:09.702461    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:16:13.182687 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.182656    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"]
Sep 09 21:16:13.183119 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.182874    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz" podUID="fa365176-6a0a-4597-bfe7-7bd70d48abaf" containerName="maas-api" containerID="cri-o://22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa" gracePeriod=30
Sep 09 21:16:13.426523 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.426498    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:16:13.559253 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.558843    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dvlf\" (UniqueName: \"kubernetes.io/projected/fa365176-6a0a-4597-bfe7-7bd70d48abaf-kube-api-access-2dvlf\") pod \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") "
Sep 09 21:16:13.559576 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.559449    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa365176-6a0a-4597-bfe7-7bd70d48abaf-openshift-service-ca\") pod \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") "
Sep 09 21:16:13.559824 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.559800    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa365176-6a0a-4597-bfe7-7bd70d48abaf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fa365176-6a0a-4597-bfe7-7bd70d48abaf" (UID: "fa365176-6a0a-4597-bfe7-7bd70d48abaf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:13.559993 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.559976    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-metrics-certs\") pod \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") "
Sep 09 21:16:13.560073 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.560026    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-maas-api-tls\") pod \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\" (UID: \"fa365176-6a0a-4597-bfe7-7bd70d48abaf\") "
Sep 09 21:16:13.560620 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.560600    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa365176-6a0a-4597-bfe7-7bd70d48abaf-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:13.562552 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.562518    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa365176-6a0a-4597-bfe7-7bd70d48abaf-kube-api-access-2dvlf" (OuterVolumeSpecName: "kube-api-access-2dvlf") pod "fa365176-6a0a-4597-bfe7-7bd70d48abaf" (UID: "fa365176-6a0a-4597-bfe7-7bd70d48abaf"). InnerVolumeSpecName "kube-api-access-2dvlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:13.562829 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.562803    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fa365176-6a0a-4597-bfe7-7bd70d48abaf" (UID: "fa365176-6a0a-4597-bfe7-7bd70d48abaf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:13.562914 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.562829    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fa365176-6a0a-4597-bfe7-7bd70d48abaf" (UID: "fa365176-6a0a-4597-bfe7-7bd70d48abaf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:13.661493 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.661456    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dvlf\" (UniqueName: \"kubernetes.io/projected/fa365176-6a0a-4597-bfe7-7bd70d48abaf-kube-api-access-2dvlf\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:13.661493 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.661487    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:13.661493 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.661496    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa365176-6a0a-4597-bfe7-7bd70d48abaf-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:13.744617 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.744584    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa365176-6a0a-4597-bfe7-7bd70d48abaf" containerID="22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa" exitCode=0
Sep 09 21:16:13.744764 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.744632    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"
Sep 09 21:16:13.744764 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.744667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz" event={"ID":"fa365176-6a0a-4597-bfe7-7bd70d48abaf","Type":"ContainerDied","Data":"22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa"}
Sep 09 21:16:13.744764 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.744724    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz" event={"ID":"fa365176-6a0a-4597-bfe7-7bd70d48abaf","Type":"ContainerDied","Data":"f1c6389e2f98bd7d02fcc4b0a2287b3a7a9189e28ee92d6b63b288864ce2e078"}
Sep 09 21:16:13.744764 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.744749    2578 scope.go:117] "RemoveContainer" containerID="22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa"
Sep 09 21:16:13.753686 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.753667    2578 scope.go:117] "RemoveContainer" containerID="22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa"
Sep 09 21:16:13.753943 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:16:13.753927    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa\": container with ID starting with 22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa not found: ID does not exist" containerID="22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa"
Sep 09 21:16:13.753991 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.753950    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa"} err="failed to get container status \"22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa\": rpc error: code = NotFound desc = could not find container \"22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa\": container with ID starting with 22d35b11ba11be23382e10414ec50eaead83995bd5c442bc0f68ed67e90b77fa not found: ID does not exist"
Sep 09 21:16:13.765676 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.765651    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"]
Sep 09 21:16:13.769475 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.769454    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-629537-5679d6c7d4-6q6gz"]
Sep 09 21:16:13.840861 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:13.840829    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa365176-6a0a-4597-bfe7-7bd70d48abaf" path="/var/lib/kubelet/pods/fa365176-6a0a-4597-bfe7-7bd70d48abaf/volumes"
Sep 09 21:16:16.697088 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:16.697051    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"]
Sep 09 21:16:17.468586 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.468554    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"]
Sep 09 21:16:17.468865 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.468853    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee70998b-6224-435f-a268-e3a57f955871" containerName="maas-api"
Sep 09 21:16:17.468912 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.468867    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee70998b-6224-435f-a268-e3a57f955871" containerName="maas-api"
Sep 09 21:16:17.468912 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.468877    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00628624-0172-46e9-ad60-83dbe4e2228f" containerName="maas-api"
Sep 09 21:16:17.468912 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.468882    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="00628624-0172-46e9-ad60-83dbe4e2228f" containerName="maas-api"
Sep 09 21:16:17.468912 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.468896    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa365176-6a0a-4597-bfe7-7bd70d48abaf" containerName="maas-api"
Sep 09 21:16:17.468912 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.468901    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa365176-6a0a-4597-bfe7-7bd70d48abaf" containerName="maas-api"
Sep 09 21:16:17.469080 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.468955    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa365176-6a0a-4597-bfe7-7bd70d48abaf" containerName="maas-api"
Sep 09 21:16:17.469080 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.468964    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee70998b-6224-435f-a268-e3a57f955871" containerName="maas-api"
Sep 09 21:16:17.469080 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.468972    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="00628624-0172-46e9-ad60-83dbe4e2228f" containerName="maas-api"
Sep 09 21:16:17.471764 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.471749    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.474376 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.474353    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d7f624-openshift-default-dockercfg-h9n7r\""
Sep 09 21:16:17.558219 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.558187    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"]
Sep 09 21:16:17.595457 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.595424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-socket\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.595607 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.595466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istiod-ca-cert\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.595607 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.595486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-token\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.595678 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.595600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-credential-socket\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.595678 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.595630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-podinfo\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.595678 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.595647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckkvf\" (UniqueName: \"kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-kube-api-access-ckkvf\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.595778 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.595677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-certs\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.595778 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.595699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-data\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.595778 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.595724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-envoy\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.697355 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.697299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-credential-socket\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.697953 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.697924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-podinfo\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.698101 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.697967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckkvf\" (UniqueName: \"kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-kube-api-access-ckkvf\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.698101 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.697994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-certs\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.698101 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.698016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-data\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.698101 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.698060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-envoy\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.698347 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.698113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-socket\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.698347 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.698160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istiod-ca-cert\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.698347 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.698207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-token\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.699391 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.699057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-data\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.699391 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.699350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-socket\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.699563 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.699544    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-certs\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.700253 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.700230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istiod-ca-cert\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.700557 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.700509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-credential-socket\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.702404 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.702134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-envoy\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.702991 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.702966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-podinfo\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.706765 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.706698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-token\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.708957 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.708928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckkvf\" (UniqueName: \"kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-kube-api-access-ckkvf\") pod \"e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") " pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:17.781215 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:17.781120    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:18.060427 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:18.060399    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"]
Sep 09 21:16:18.062132 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:16:18.062104    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc9af60_1868_4d0b_af7f_7d4efe9914a7.slice/crio-28c877a3b272cd4af107bc796574693b4b71f7560717ebf34146ba236540c63a WatchSource:0}: Error finding container 28c877a3b272cd4af107bc796574693b4b71f7560717ebf34146ba236540c63a: Status 404 returned error can't find the container with id 28c877a3b272cd4af107bc796574693b4b71f7560717ebf34146ba236540c63a
Sep 09 21:16:18.064286 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:18.064252    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:16:18.064361 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:18.064319    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:16:18.064361 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:18.064346    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:16:18.772310 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:18.772274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" event={"ID":"8fc9af60-1868-4d0b-af7f-7d4efe9914a7","Type":"ContainerStarted","Data":"5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551"}
Sep 09 21:16:18.772697 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:18.772333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" event={"ID":"8fc9af60-1868-4d0b-af7f-7d4efe9914a7","Type":"ContainerStarted","Data":"28c877a3b272cd4af107bc796574693b4b71f7560717ebf34146ba236540c63a"}
Sep 09 21:16:18.781229 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:18.781201    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:18.795774 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:18.795731    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" podStartSLOduration=1.795717605 podStartE2EDuration="1.795717605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:18.793926557 +0000 UTC m=+1655.448706739" watchObservedRunningTime="2026-09-09 21:16:18.795717605 +0000 UTC m=+1655.450497763"
Sep 09 21:16:19.786401 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:19.786370    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:20.047076 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:20.047001    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"]
Sep 09 21:16:20.047315 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:20.047283    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy" containerID="cri-o://c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176" gracePeriod=30
Sep 09 21:16:20.781347 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:20.781314    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:20.782289 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:20.782266    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:21.954808 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:21.954778    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"]
Sep 09 21:16:21.957861 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:21.957838    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:21.960168 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:21.960152    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d7f624-serving-cert\""
Sep 09 21:16:21.969322 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:21.969300    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"]
Sep 09 21:16:22.037850 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.037816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9j5t\" (UniqueName: \"kubernetes.io/projected/2f850012-11d6-4b27-92e7-ecb5f124e461-kube-api-access-z9j5t\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.037850 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.037854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f850012-11d6-4b27-92e7-ecb5f124e461-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.038040 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.037885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.038040 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.037963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-metrics-certs\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.138693 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.138658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9j5t\" (UniqueName: \"kubernetes.io/projected/2f850012-11d6-4b27-92e7-ecb5f124e461-kube-api-access-z9j5t\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.138851 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.138703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f850012-11d6-4b27-92e7-ecb5f124e461-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.138851 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.138737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.138851 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.138770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-metrics-certs\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.138958 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:16:22.138877    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-serving-cert: secret "maas-api-e2e-ipp-a-d7f624-serving-cert" not found
Sep 09 21:16:22.138958 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:16:22.138940    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-maas-api-tls podName:2f850012-11d6-4b27-92e7-ecb5f124e461 nodeName:}" failed. No retries permitted until 2026-09-09 21:16:22.638923698 +0000 UTC m=+1659.293703855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-maas-api-tls") pod "maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x" (UID: "2f850012-11d6-4b27-92e7-ecb5f124e461") : secret "maas-api-e2e-ipp-a-d7f624-serving-cert" not found
Sep 09 21:16:22.139453 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.139429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f850012-11d6-4b27-92e7-ecb5f124e461-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.140986 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.140961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-metrics-certs\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.149154 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.149135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9j5t\" (UniqueName: \"kubernetes.io/projected/2f850012-11d6-4b27-92e7-ecb5f124e461-kube-api-access-z9j5t\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.643321 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.643288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.645560 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.645539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:22.869960 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:22.869915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:23.142306 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:23.142201    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"]
Sep 09 21:16:23.144454 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:16:23.144429    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f850012_11d6_4b27_92e7_ecb5f124e461.slice/crio-e5b82611c50a03e82936b2b2d26f733c9deca7d919b2a8a0e8aaffbafc377a35 WatchSource:0}: Error finding container e5b82611c50a03e82936b2b2d26f733c9deca7d919b2a8a0e8aaffbafc377a35: Status 404 returned error can't find the container with id e5b82611c50a03e82936b2b2d26f733c9deca7d919b2a8a0e8aaffbafc377a35
Sep 09 21:16:23.145792 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:23.145773    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:16:23.405863 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:23.405832    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 21:16:23.406016 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:23.405887    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:16:23.795712 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:23.795640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x" event={"ID":"2f850012-11d6-4b27-92e7-ecb5f124e461","Type":"ContainerStarted","Data":"e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca"}
Sep 09 21:16:23.795712 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:23.795688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x" event={"ID":"2f850012-11d6-4b27-92e7-ecb5f124e461","Type":"ContainerStarted","Data":"e5b82611c50a03e82936b2b2d26f733c9deca7d919b2a8a0e8aaffbafc377a35"}
Sep 09 21:16:23.795879 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:23.795783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:25.299783 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.299759    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:16:25.318303 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.318196    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x" podStartSLOduration=3.890659473 podStartE2EDuration="4.318161021s" totalImagesPullingTime="427.501548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:23.813296462 +0000 UTC m=+1660.468076621" watchObservedRunningTime="2026-09-09 21:16:25.318161021 +0000 UTC m=+1661.972941179"
Sep 09 21:16:25.468825 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.468778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-credential-socket\") pod \"4262be42-a2d0-4cce-a596-26c57f2aa11f\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") "
Sep 09 21:16:25.469016 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.468844    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-data\") pod \"4262be42-a2d0-4cce-a596-26c57f2aa11f\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") "
Sep 09 21:16:25.469016 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.468865    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-envoy\") pod \"4262be42-a2d0-4cce-a596-26c57f2aa11f\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") "
Sep 09 21:16:25.469016 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.468895    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-podinfo\") pod \"4262be42-a2d0-4cce-a596-26c57f2aa11f\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") "
Sep 09 21:16:25.469016 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.468911    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-socket\") pod \"4262be42-a2d0-4cce-a596-26c57f2aa11f\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") "
Sep 09 21:16:25.469016 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.468951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzwch\" (UniqueName: \"kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-kube-api-access-hzwch\") pod \"4262be42-a2d0-4cce-a596-26c57f2aa11f\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") "
Sep 09 21:16:25.469016 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.468966    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4262be42-a2d0-4cce-a596-26c57f2aa11f-istiod-ca-cert\") pod \"4262be42-a2d0-4cce-a596-26c57f2aa11f\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") "
Sep 09 21:16:25.469016 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.468990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-token\") pod \"4262be42-a2d0-4cce-a596-26c57f2aa11f\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") "
Sep 09 21:16:25.469424 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.469028    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-certs\") pod \"4262be42-a2d0-4cce-a596-26c57f2aa11f\" (UID: \"4262be42-a2d0-4cce-a596-26c57f2aa11f\") "
Sep 09 21:16:25.469424 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.469085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-data" (OuterVolumeSpecName: "istio-data") pod "4262be42-a2d0-4cce-a596-26c57f2aa11f" (UID: "4262be42-a2d0-4cce-a596-26c57f2aa11f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:25.469424 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.469104    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4262be42-a2d0-4cce-a596-26c57f2aa11f" (UID: "4262be42-a2d0-4cce-a596-26c57f2aa11f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:25.469424 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.469320    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4262be42-a2d0-4cce-a596-26c57f2aa11f" (UID: "4262be42-a2d0-4cce-a596-26c57f2aa11f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:25.469424 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.469377    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:25.469424 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.469396    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:25.469746 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.469562    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4262be42-a2d0-4cce-a596-26c57f2aa11f" (UID: "4262be42-a2d0-4cce-a596-26c57f2aa11f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:25.469746 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.469631    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4262be42-a2d0-4cce-a596-26c57f2aa11f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4262be42-a2d0-4cce-a596-26c57f2aa11f" (UID: "4262be42-a2d0-4cce-a596-26c57f2aa11f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:25.471221 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.471164    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-token" (OuterVolumeSpecName: "istio-token") pod "4262be42-a2d0-4cce-a596-26c57f2aa11f" (UID: "4262be42-a2d0-4cce-a596-26c57f2aa11f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:25.471337 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.471314    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-kube-api-access-hzwch" (OuterVolumeSpecName: "kube-api-access-hzwch") pod "4262be42-a2d0-4cce-a596-26c57f2aa11f" (UID: "4262be42-a2d0-4cce-a596-26c57f2aa11f"). InnerVolumeSpecName "kube-api-access-hzwch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:25.471539 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.471503    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4262be42-a2d0-4cce-a596-26c57f2aa11f" (UID: "4262be42-a2d0-4cce-a596-26c57f2aa11f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:25.471539 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.471503    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4262be42-a2d0-4cce-a596-26c57f2aa11f" (UID: "4262be42-a2d0-4cce-a596-26c57f2aa11f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:16:25.569879 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.569803    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:25.569879 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.569828    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:25.569879 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.569837    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:25.569879 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.569847    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzwch\" (UniqueName: \"kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-kube-api-access-hzwch\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:25.569879 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.569855    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4262be42-a2d0-4cce-a596-26c57f2aa11f-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:25.569879 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.569863    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4262be42-a2d0-4cce-a596-26c57f2aa11f-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:25.569879 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.569870    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4262be42-a2d0-4cce-a596-26c57f2aa11f-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:25.806744 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.806710    2578 generic.go:356] "Generic (PLEG): container finished" podID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerID="c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176" exitCode=0
Sep 09 21:16:25.806906 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.806759    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"
Sep 09 21:16:25.806906 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.806798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" event={"ID":"4262be42-a2d0-4cce-a596-26c57f2aa11f","Type":"ContainerDied","Data":"c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176"}
Sep 09 21:16:25.806906 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.806852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6" event={"ID":"4262be42-a2d0-4cce-a596-26c57f2aa11f","Type":"ContainerDied","Data":"d721fc00053d3562cfc58946052d1e50a8be31a38bb7d4835f7815eb40864de0"}
Sep 09 21:16:25.806906 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.806876    2578 scope.go:117] "RemoveContainer" containerID="c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176"
Sep 09 21:16:25.816226 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.816205    2578 scope.go:117] "RemoveContainer" containerID="5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19"
Sep 09 21:16:25.825587 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.825559    2578 scope.go:117] "RemoveContainer" containerID="c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176"
Sep 09 21:16:25.826459 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:16:25.826428    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176\": container with ID starting with c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176 not found: ID does not exist" containerID="c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176"
Sep 09 21:16:25.826585 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.826477    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176"} err="failed to get container status \"c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176\": rpc error: code = NotFound desc = could not find container \"c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176\": container with ID starting with c183865d3e54a62f411418e1aeea2c93da8117e88eaabd0da33dfc7b80b08176 not found: ID does not exist"
Sep 09 21:16:25.826585 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.826502    2578 scope.go:117] "RemoveContainer" containerID="5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19"
Sep 09 21:16:25.826973 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:16:25.826951    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19\": container with ID starting with 5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19 not found: ID does not exist" containerID="5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19"
Sep 09 21:16:25.827072 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.826978    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19"} err="failed to get container status \"5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19\": rpc error: code = NotFound desc = could not find container \"5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19\": container with ID starting with 5293737d2306a81b6c2eb73585717bff6afcaff4cbac1442c351ab8e7335ed19 not found: ID does not exist"
Sep 09 21:16:25.827755 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.827737    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"]
Sep 09 21:16:25.831500 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.831479    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-629537-openshift-default-7f6dff4dc4-jfgn6"]
Sep 09 21:16:25.840267 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:25.840248    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" path="/var/lib/kubelet/pods/4262be42-a2d0-4cce-a596-26c57f2aa11f/volumes"
Sep 09 21:16:26.469581 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:26.469546    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:16:29.805610 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:29.805578    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:16:31.720790 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:31.720742    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv" podUID="f9031446-060d-406f-94ba-c5f741728b5a" containerName="main" containerID="cri-o://f3fb752817125a933eed54d822ad82c7979edfab1fa46523f557920e21851f3d" gracePeriod=45
Sep 09 21:16:31.846923 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:31.846849    2578 generic.go:356] "Generic (PLEG): container finished" podID="f9031446-060d-406f-94ba-c5f741728b5a" containerID="f3fb752817125a933eed54d822ad82c7979edfab1fa46523f557920e21851f3d" exitCode=0
Sep 09 21:16:31.847007 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:31.846927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv" event={"ID":"f9031446-060d-406f-94ba-c5f741728b5a","Type":"ContainerDied","Data":"f3fb752817125a933eed54d822ad82c7979edfab1fa46523f557920e21851f3d"}
Sep 09 21:16:31.991381 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:31.991357    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:16:32.144322 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.144284    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9031446-060d-406f-94ba-c5f741728b5a-tls-certs\") pod \"f9031446-060d-406f-94ba-c5f741728b5a\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") "
Sep 09 21:16:32.144513 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.144337    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-model-cache\") pod \"f9031446-060d-406f-94ba-c5f741728b5a\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") "
Sep 09 21:16:32.144513 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.144386    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcjg5\" (UniqueName: \"kubernetes.io/projected/f9031446-060d-406f-94ba-c5f741728b5a-kube-api-access-jcjg5\") pod \"f9031446-060d-406f-94ba-c5f741728b5a\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") "
Sep 09 21:16:32.144513 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.144429    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-dshm\") pod \"f9031446-060d-406f-94ba-c5f741728b5a\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") "
Sep 09 21:16:32.144513 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.144459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-home\") pod \"f9031446-060d-406f-94ba-c5f741728b5a\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") "
Sep 09 21:16:32.144513 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.144499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-tmp-dir\") pod \"f9031446-060d-406f-94ba-c5f741728b5a\" (UID: \"f9031446-060d-406f-94ba-c5f741728b5a\") "
Sep 09 21:16:32.144776 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.144634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9031446-060d-406f-94ba-c5f741728b5a" (UID: "f9031446-060d-406f-94ba-c5f741728b5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:32.144776 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.144691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-home" (OuterVolumeSpecName: "home") pod "f9031446-060d-406f-94ba-c5f741728b5a" (UID: "f9031446-060d-406f-94ba-c5f741728b5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:32.144884 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.144810    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-home\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:32.144884 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.144817    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9031446-060d-406f-94ba-c5f741728b5a" (UID: "f9031446-060d-406f-94ba-c5f741728b5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:32.144884 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.144823    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-model-cache\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:32.146603 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.146571    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-dshm" (OuterVolumeSpecName: "dshm") pod "f9031446-060d-406f-94ba-c5f741728b5a" (UID: "f9031446-060d-406f-94ba-c5f741728b5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:32.146710 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.146629    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9031446-060d-406f-94ba-c5f741728b5a-kube-api-access-jcjg5" (OuterVolumeSpecName: "kube-api-access-jcjg5") pod "f9031446-060d-406f-94ba-c5f741728b5a" (UID: "f9031446-060d-406f-94ba-c5f741728b5a"). InnerVolumeSpecName "kube-api-access-jcjg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:32.146710 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.146639    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9031446-060d-406f-94ba-c5f741728b5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9031446-060d-406f-94ba-c5f741728b5a" (UID: "f9031446-060d-406f-94ba-c5f741728b5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:32.245673 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.245603    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcjg5\" (UniqueName: \"kubernetes.io/projected/f9031446-060d-406f-94ba-c5f741728b5a-kube-api-access-jcjg5\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:32.245673 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.245627    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-dshm\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:32.245673 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.245637    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9031446-060d-406f-94ba-c5f741728b5a-tmp-dir\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:32.245673 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.245647    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9031446-060d-406f-94ba-c5f741728b5a-tls-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:32.852479 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.852449    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"
Sep 09 21:16:32.852948 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.852447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv" event={"ID":"f9031446-060d-406f-94ba-c5f741728b5a","Type":"ContainerDied","Data":"41be8cb242a9bef0c2fdc7f194e6ab3ca12b7d05b79c3b411e99c6c4d57a76d8"}
Sep 09 21:16:32.852948 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.852582    2578 scope.go:117] "RemoveContainer" containerID="f3fb752817125a933eed54d822ad82c7979edfab1fa46523f557920e21851f3d"
Sep 09 21:16:32.877921 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.877893    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"]
Sep 09 21:16:32.881575 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:32.881551    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/rate-test-model-8db5a7-kserve-76cf4fb579-x9xpv"]
Sep 09 21:16:33.842930 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:33.842901    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9031446-060d-406f-94ba-c5f741728b5a" path="/var/lib/kubelet/pods/f9031446-060d-406f-94ba-c5f741728b5a/volumes"
Sep 09 21:16:36.449631 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:36.449597    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:35096->10.134.0.53:15021: read: connection reset by peer" start-of-body=
Sep 09 21:16:36.450105 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:36.449661    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:35096->10.134.0.53:15021: read: connection reset by peer"
Sep 09 21:16:36.879014 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:36.878983    2578 generic.go:356] "Generic (PLEG): container finished" podID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerID="5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551" exitCode=0
Sep 09 21:16:36.879190 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:36.879054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" event={"ID":"8fc9af60-1868-4d0b-af7f-7d4efe9914a7","Type":"ContainerDied","Data":"5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551"}
Sep 09 21:16:36.879469 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:36.879455    2578 scope.go:117] "RemoveContainer" containerID="5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551"
Sep 09 21:16:36.880525 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:36.880500    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:16:36.880591 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:36.880573    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:16:36.880628 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:36.880602    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:16:37.886760 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:37.886725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" event={"ID":"8fc9af60-1868-4d0b-af7f-7d4efe9914a7","Type":"ContainerStarted","Data":"f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343"}
Sep 09 21:16:38.160796 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:38.160713    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"]
Sep 09 21:16:38.160978 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:38.160953    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy" containerID="cri-o://2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5" gracePeriod=30
Sep 09 21:16:38.781739 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:38.781703    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:38.781932 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:38.781907    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 09 21:16:38.781988 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:38.781960    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 09 21:16:39.782001 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:39.781964    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 09 21:16:39.782398 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:39.782036    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 09 21:16:40.604748 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:40.604715    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:16:40.782006 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:40.781967    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 09 21:16:40.782398 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:40.782025    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 09 21:16:41.781499 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:41.781457    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 09 21:16:41.781688 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:41.781530    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 09 21:16:42.782509 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:42.782471    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 09 21:16:42.782885 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:42.782530    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 09 21:16:43.420016 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.419989    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:16:43.443923 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.443766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-certs\") pod \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") "
Sep 09 21:16:43.443923 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.443816    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49kfq\" (UniqueName: \"kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-kube-api-access-49kfq\") pod \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") "
Sep 09 21:16:43.443923 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.443853    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-envoy\") pod \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") "
Sep 09 21:16:43.443923 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.443883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-socket\") pod \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") "
Sep 09 21:16:43.443923 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.443919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-data\") pod \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") "
Sep 09 21:16:43.445524 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.443966    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-credential-socket\") pod \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") "
Sep 09 21:16:43.445524 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.444014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-token\") pod \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") "
Sep 09 21:16:43.445524 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.444037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-podinfo\") pod \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") "
Sep 09 21:16:43.445524 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.444100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istiod-ca-cert\") pod \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\" (UID: \"ac18b9ad-1250-4a5f-8ec9-00234524b7f8\") "
Sep 09 21:16:43.445524 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.444451    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac18b9ad-1250-4a5f-8ec9-00234524b7f8" (UID: "ac18b9ad-1250-4a5f-8ec9-00234524b7f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:43.445524 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.444716    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac18b9ad-1250-4a5f-8ec9-00234524b7f8" (UID: "ac18b9ad-1250-4a5f-8ec9-00234524b7f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:43.445524 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.444733    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac18b9ad-1250-4a5f-8ec9-00234524b7f8" (UID: "ac18b9ad-1250-4a5f-8ec9-00234524b7f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:43.445524 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.444778    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac18b9ad-1250-4a5f-8ec9-00234524b7f8" (UID: "ac18b9ad-1250-4a5f-8ec9-00234524b7f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:43.445524 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.444976    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac18b9ad-1250-4a5f-8ec9-00234524b7f8" (UID: "ac18b9ad-1250-4a5f-8ec9-00234524b7f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:43.449089 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.447789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac18b9ad-1250-4a5f-8ec9-00234524b7f8" (UID: "ac18b9ad-1250-4a5f-8ec9-00234524b7f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:43.449089 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.448438    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-kube-api-access-49kfq" (OuterVolumeSpecName: "kube-api-access-49kfq") pod "ac18b9ad-1250-4a5f-8ec9-00234524b7f8" (UID: "ac18b9ad-1250-4a5f-8ec9-00234524b7f8"). InnerVolumeSpecName "kube-api-access-49kfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:43.449089 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.448460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac18b9ad-1250-4a5f-8ec9-00234524b7f8" (UID: "ac18b9ad-1250-4a5f-8ec9-00234524b7f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:16:43.449089 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.448708    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac18b9ad-1250-4a5f-8ec9-00234524b7f8" (UID: "ac18b9ad-1250-4a5f-8ec9-00234524b7f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:43.545448 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.545414    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:43.545448 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.545440    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:43.545448 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.545449    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:43.545685 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.545457    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:43.545685 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.545467    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:43.545685 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.545476    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:43.545685 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.545484    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:43.545685 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.545491    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49kfq\" (UniqueName: \"kubernetes.io/projected/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-kube-api-access-49kfq\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:43.545685 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.545499    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac18b9ad-1250-4a5f-8ec9-00234524b7f8-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:16:43.785502 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.785422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:43.785957 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.785804    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:43.786406 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.786388    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:16:43.918751 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.918723    2578 generic.go:356] "Generic (PLEG): container finished" podID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerID="2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5" exitCode=0
Sep 09 21:16:43.918928 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.918768    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"
Sep 09 21:16:43.918928 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.918810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" event={"ID":"ac18b9ad-1250-4a5f-8ec9-00234524b7f8","Type":"ContainerDied","Data":"2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5"}
Sep 09 21:16:43.918928 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.918867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g" event={"ID":"ac18b9ad-1250-4a5f-8ec9-00234524b7f8","Type":"ContainerDied","Data":"957f0f4dbd0f4521169cf143cb7fbc8ed1f2b60818ccd06542fcecdb046d5555"}
Sep 09 21:16:43.918928 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.918890    2578 scope.go:117] "RemoveContainer" containerID="2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5"
Sep 09 21:16:43.926955 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.926941    2578 scope.go:117] "RemoveContainer" containerID="685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e"
Sep 09 21:16:43.934522 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.934502    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"]
Sep 09 21:16:43.934728 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.934716    2578 scope.go:117] "RemoveContainer" containerID="2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5"
Sep 09 21:16:43.935003 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:16:43.934977    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5\": container with ID starting with 2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5 not found: ID does not exist" containerID="2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5"
Sep 09 21:16:43.935070 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.935005    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5"} err="failed to get container status \"2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5\": rpc error: code = NotFound desc = could not find container \"2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5\": container with ID starting with 2b39f14abc781fd1a11c66d51dca8b4bcd012d03457b7fa86adedd15d0a731b5 not found: ID does not exist"
Sep 09 21:16:43.935070 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.935022    2578 scope.go:117] "RemoveContainer" containerID="685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e"
Sep 09 21:16:43.935292 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:16:43.935269    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e\": container with ID starting with 685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e not found: ID does not exist" containerID="685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e"
Sep 09 21:16:43.935342 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.935291    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e"} err="failed to get container status \"685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e\": rpc error: code = NotFound desc = could not find container \"685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e\": container with ID starting with 685791fdf6aa8e2097c27959833b57fdcecd338c1052e52719f19eb4881f0e1e not found: ID does not exist"
Sep 09 21:16:43.940986 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:43.940966    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-459158-openshift-default-898564cd4-qcw6g"]
Sep 09 21:16:45.841089 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:45.841051    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" path="/var/lib/kubelet/pods/ac18b9ad-1250-4a5f-8ec9-00234524b7f8/volumes"
Sep 09 21:16:51.441386 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441354    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"]
Sep 09 21:16:51.441737 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441675    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy"
Sep 09 21:16:51.441737 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441686    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy"
Sep 09 21:16:51.441737 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441700    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy"
Sep 09 21:16:51.441737 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441705    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy"
Sep 09 21:16:51.441737 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441716    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9031446-060d-406f-94ba-c5f741728b5a" containerName="main"
Sep 09 21:16:51.441737 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441721    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9031446-060d-406f-94ba-c5f741728b5a" containerName="main"
Sep 09 21:16:51.441737 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441727    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy"
Sep 09 21:16:51.441737 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441732    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy"
Sep 09 21:16:51.441994 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441782    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9031446-060d-406f-94ba-c5f741728b5a" containerName="main"
Sep 09 21:16:51.441994 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441790    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy"
Sep 09 21:16:51.441994 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.441796    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy"
Sep 09 21:16:51.444830 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.444811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:51.449465 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.449443    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d7dd56-serving-cert\""
Sep 09 21:16:51.470104 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.470077    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"]
Sep 09 21:16:51.512706 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.512673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:51.512877 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.512715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a2b388-a502-4d55-b484-171a60cae7d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:51.512877 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.512740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlzh4\" (UniqueName: \"kubernetes.io/projected/d3a2b388-a502-4d55-b484-171a60cae7d6-kube-api-access-xlzh4\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:51.512877 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.512840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-metrics-certs\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:51.613954 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.613920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:51.614126 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.613963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a2b388-a502-4d55-b484-171a60cae7d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:51.614126 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.613984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlzh4\" (UniqueName: \"kubernetes.io/projected/d3a2b388-a502-4d55-b484-171a60cae7d6-kube-api-access-xlzh4\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:51.614126 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.614059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-metrics-certs\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:51.614314 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:16:51.614163    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-serving-cert: secret "maas-api-e2e-ipp-b-d7dd56-serving-cert" not found
Sep 09 21:16:51.614314 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:16:51.614266    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-maas-api-tls podName:d3a2b388-a502-4d55-b484-171a60cae7d6 nodeName:}" failed. No retries permitted until 2026-09-09 21:16:52.114245291 +0000 UTC m=+1688.769025443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-maas-api-tls") pod "maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2" (UID: "d3a2b388-a502-4d55-b484-171a60cae7d6") : secret "maas-api-e2e-ipp-b-d7dd56-serving-cert" not found
Sep 09 21:16:51.614581 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.614555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a2b388-a502-4d55-b484-171a60cae7d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:51.616377 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.616358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-metrics-certs\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:51.628278 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:51.628252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlzh4\" (UniqueName: \"kubernetes.io/projected/d3a2b388-a502-4d55-b484-171a60cae7d6-kube-api-access-xlzh4\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:52.118918 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:52.118886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:52.121118 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:52.121099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:52.354311 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:52.354273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:52.630224 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:52.630194    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"]
Sep 09 21:16:52.631839 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:16:52.631813    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a2b388_a502_4d55_b484_171a60cae7d6.slice/crio-9f22c72cb37db70adbb49577d4aeed624ef6304dc49d7e55fd12f7b449a94c68 WatchSource:0}: Error finding container 9f22c72cb37db70adbb49577d4aeed624ef6304dc49d7e55fd12f7b449a94c68: Status 404 returned error can't find the container with id 9f22c72cb37db70adbb49577d4aeed624ef6304dc49d7e55fd12f7b449a94c68
Sep 09 21:16:52.957588 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:52.957565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2" event={"ID":"d3a2b388-a502-4d55-b484-171a60cae7d6","Type":"ContainerStarted","Data":"9f22c72cb37db70adbb49577d4aeed624ef6304dc49d7e55fd12f7b449a94c68"}
Sep 09 21:16:53.964001 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:53.963959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2" event={"ID":"d3a2b388-a502-4d55-b484-171a60cae7d6","Type":"ContainerStarted","Data":"ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da"}
Sep 09 21:16:53.964477 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:53.964018    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:16:53.980528 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:53.980409    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2" podStartSLOduration=2.660395123 podStartE2EDuration="2.980392825s" totalImagesPullingTime="319.997702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:53.980017557 +0000 UTC m=+1690.634797707" watchObservedRunningTime="2026-09-09 21:16:53.980392825 +0000 UTC m=+1690.635172984"
Sep 09 21:16:56.212753 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:56.212717    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:16:59.973578 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:16:59.973542    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:17:38.820721 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:17:38.820684    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:17:52.812561 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:17:52.812530    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:18:32.806561 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.806526    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"]
Sep 09 21:18:32.807156 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.807060    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy"
Sep 09 21:18:32.807156 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.807079    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy"
Sep 09 21:18:32.807344 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.807187    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4262be42-a2d0-4cce-a596-26c57f2aa11f" containerName="istio-proxy"
Sep 09 21:18:32.807344 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.807201    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac18b9ad-1250-4a5f-8ec9-00234524b7f8" containerName="istio-proxy"
Sep 09 21:18:32.810258 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.810235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:32.812424 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.812391    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-8d0f62-serving-cert\""
Sep 09 21:18:32.823709 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.823685    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"]
Sep 09 21:18:32.890069 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.890042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:32.890226 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.890090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103c73a4-78fd-455c-80be-1500534369ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:32.890328 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.890312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:32.890392 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.890380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv59w\" (UniqueName: \"kubernetes.io/projected/103c73a4-78fd-455c-80be-1500534369ff-kube-api-access-rv59w\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:32.991392 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.991356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:32.991581 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.991396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103c73a4-78fd-455c-80be-1500534369ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:32.991581 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.991437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:32.991581 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.991477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv59w\" (UniqueName: \"kubernetes.io/projected/103c73a4-78fd-455c-80be-1500534369ff-kube-api-access-rv59w\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:32.991581 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:18:32.991510    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-serving-cert: secret "maas-api-e2e-ipp-cleanup-8d0f62-serving-cert" not found
Sep 09 21:18:32.991776 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:18:32.991595    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-maas-api-tls podName:103c73a4-78fd-455c-80be-1500534369ff nodeName:}" failed. No retries permitted until 2026-09-09 21:18:33.491577822 +0000 UTC m=+1790.146357963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w" (UID: "103c73a4-78fd-455c-80be-1500534369ff") : secret "maas-api-e2e-ipp-cleanup-8d0f62-serving-cert" not found
Sep 09 21:18:32.992089 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.992068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103c73a4-78fd-455c-80be-1500534369ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:32.993923 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:32.993902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:33.000523 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:33.000504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv59w\" (UniqueName: \"kubernetes.io/projected/103c73a4-78fd-455c-80be-1500534369ff-kube-api-access-rv59w\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:33.495093 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:33.495057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:33.497391 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:33.497362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:33.720565 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:33.720529    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:34.001374 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:34.001347    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"]
Sep 09 21:18:34.003079 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:18:34.003052    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103c73a4_78fd_455c_80be_1500534369ff.slice/crio-eba026c5ec57ab79785491ef5d3e83615ba1f03e0b94d990908a4283107961cd WatchSource:0}: Error finding container eba026c5ec57ab79785491ef5d3e83615ba1f03e0b94d990908a4283107961cd: Status 404 returned error can't find the container with id eba026c5ec57ab79785491ef5d3e83615ba1f03e0b94d990908a4283107961cd
Sep 09 21:18:34.394777 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:34.394742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w" event={"ID":"103c73a4-78fd-455c-80be-1500534369ff","Type":"ContainerStarted","Data":"eba026c5ec57ab79785491ef5d3e83615ba1f03e0b94d990908a4283107961cd"}
Sep 09 21:18:35.400794 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:35.400755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w" event={"ID":"103c73a4-78fd-455c-80be-1500534369ff","Type":"ContainerStarted","Data":"6d5dffde0a21fa2b1869039580a7ff69172a316363775649380435e40dff5e80"}
Sep 09 21:18:35.401199 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:35.400857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:18:35.434254 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:35.434205    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w" podStartSLOduration=3.022999052 podStartE2EDuration="3.434188469s" totalImagesPullingTime="411.189417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:18:35.433026691 +0000 UTC m=+1792.087806843" watchObservedRunningTime="2026-09-09 21:18:35.434188469 +0000 UTC m=+1792.088968621"
Sep 09 21:18:41.307890 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:41.307809    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:18:41.411161 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:18:41.411133    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:19:03.347394 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.347361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"]
Sep 09 21:19:03.347824 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.347587    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w" podUID="103c73a4-78fd-455c-80be-1500534369ff" containerName="maas-api" containerID="cri-o://6d5dffde0a21fa2b1869039580a7ff69172a316363775649380435e40dff5e80" gracePeriod=30
Sep 09 21:19:03.535031 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.534995    2578 generic.go:356] "Generic (PLEG): container finished" podID="103c73a4-78fd-455c-80be-1500534369ff" containerID="6d5dffde0a21fa2b1869039580a7ff69172a316363775649380435e40dff5e80" exitCode=0
Sep 09 21:19:03.535308 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.535030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w" event={"ID":"103c73a4-78fd-455c-80be-1500534369ff","Type":"ContainerDied","Data":"6d5dffde0a21fa2b1869039580a7ff69172a316363775649380435e40dff5e80"}
Sep 09 21:19:03.599379 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.599326    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:19:03.651413 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.651371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-metrics-certs\") pod \"103c73a4-78fd-455c-80be-1500534369ff\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") "
Sep 09 21:19:03.651413 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.651419    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv59w\" (UniqueName: \"kubernetes.io/projected/103c73a4-78fd-455c-80be-1500534369ff-kube-api-access-rv59w\") pod \"103c73a4-78fd-455c-80be-1500534369ff\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") "
Sep 09 21:19:03.651673 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.651442    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103c73a4-78fd-455c-80be-1500534369ff-openshift-service-ca\") pod \"103c73a4-78fd-455c-80be-1500534369ff\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") "
Sep 09 21:19:03.651673 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.651486    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-maas-api-tls\") pod \"103c73a4-78fd-455c-80be-1500534369ff\" (UID: \"103c73a4-78fd-455c-80be-1500534369ff\") "
Sep 09 21:19:03.651850 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.651823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103c73a4-78fd-455c-80be-1500534369ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "103c73a4-78fd-455c-80be-1500534369ff" (UID: "103c73a4-78fd-455c-80be-1500534369ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:19:03.653553 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.653531    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "103c73a4-78fd-455c-80be-1500534369ff" (UID: "103c73a4-78fd-455c-80be-1500534369ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:19:03.653634 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.653563    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "103c73a4-78fd-455c-80be-1500534369ff" (UID: "103c73a4-78fd-455c-80be-1500534369ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:19:03.653709 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.653694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103c73a4-78fd-455c-80be-1500534369ff-kube-api-access-rv59w" (OuterVolumeSpecName: "kube-api-access-rv59w") pod "103c73a4-78fd-455c-80be-1500534369ff" (UID: "103c73a4-78fd-455c-80be-1500534369ff"). InnerVolumeSpecName "kube-api-access-rv59w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:19:03.752596 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.752541    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:19:03.752596 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.752593    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv59w\" (UniqueName: \"kubernetes.io/projected/103c73a4-78fd-455c-80be-1500534369ff-kube-api-access-rv59w\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:19:03.752882 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.752609    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103c73a4-78fd-455c-80be-1500534369ff-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:19:03.752882 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:03.752624    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103c73a4-78fd-455c-80be-1500534369ff-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:19:04.540938 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:04.540904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w" event={"ID":"103c73a4-78fd-455c-80be-1500534369ff","Type":"ContainerDied","Data":"eba026c5ec57ab79785491ef5d3e83615ba1f03e0b94d990908a4283107961cd"}
Sep 09 21:19:04.541397 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:04.540972    2578 scope.go:117] "RemoveContainer" containerID="6d5dffde0a21fa2b1869039580a7ff69172a316363775649380435e40dff5e80"
Sep 09 21:19:04.541397 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:04.540919    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"
Sep 09 21:19:04.562397 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:04.562369    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"]
Sep 09 21:19:04.564186 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:04.564151    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8d0f62-7f8cb68b79-hl89w"]
Sep 09 21:19:05.840855 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:05.840819    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103c73a4-78fd-455c-80be-1500534369ff" path="/var/lib/kubelet/pods/103c73a4-78fd-455c-80be-1500534369ff/volumes"
Sep 09 21:19:07.613456 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:07.613419    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:19:27.413287 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:27.413249    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:19:29.557094 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.557058    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"]
Sep 09 21:19:29.557489 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.557270    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x" podUID="2f850012-11d6-4b27-92e7-ecb5f124e461" containerName="maas-api" containerID="cri-o://e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca" gracePeriod=30
Sep 09 21:19:29.796076 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.796053    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:19:29.873288 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.873259    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f850012-11d6-4b27-92e7-ecb5f124e461-openshift-service-ca\") pod \"2f850012-11d6-4b27-92e7-ecb5f124e461\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") "
Sep 09 21:19:29.873486 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.873303    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-maas-api-tls\") pod \"2f850012-11d6-4b27-92e7-ecb5f124e461\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") "
Sep 09 21:19:29.873486 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.873327    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-metrics-certs\") pod \"2f850012-11d6-4b27-92e7-ecb5f124e461\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") "
Sep 09 21:19:29.873486 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.873450    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9j5t\" (UniqueName: \"kubernetes.io/projected/2f850012-11d6-4b27-92e7-ecb5f124e461-kube-api-access-z9j5t\") pod \"2f850012-11d6-4b27-92e7-ecb5f124e461\" (UID: \"2f850012-11d6-4b27-92e7-ecb5f124e461\") "
Sep 09 21:19:29.873671 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.873635    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f850012-11d6-4b27-92e7-ecb5f124e461-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f850012-11d6-4b27-92e7-ecb5f124e461" (UID: "2f850012-11d6-4b27-92e7-ecb5f124e461"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:19:29.875448 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.875419    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f850012-11d6-4b27-92e7-ecb5f124e461" (UID: "2f850012-11d6-4b27-92e7-ecb5f124e461"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:19:29.875552 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.875490    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f850012-11d6-4b27-92e7-ecb5f124e461-kube-api-access-z9j5t" (OuterVolumeSpecName: "kube-api-access-z9j5t") pod "2f850012-11d6-4b27-92e7-ecb5f124e461" (UID: "2f850012-11d6-4b27-92e7-ecb5f124e461"). InnerVolumeSpecName "kube-api-access-z9j5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:19:29.875552 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.875497    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2f850012-11d6-4b27-92e7-ecb5f124e461" (UID: "2f850012-11d6-4b27-92e7-ecb5f124e461"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:19:29.974187 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.974126    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f850012-11d6-4b27-92e7-ecb5f124e461-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:19:29.974187 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.974182    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:19:29.974388 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.974197    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f850012-11d6-4b27-92e7-ecb5f124e461-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:19:29.974388 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:29.974208    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9j5t\" (UniqueName: \"kubernetes.io/projected/2f850012-11d6-4b27-92e7-ecb5f124e461-kube-api-access-z9j5t\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:19:30.661649 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:30.661611    2578 generic.go:356] "Generic (PLEG): container finished" podID="2f850012-11d6-4b27-92e7-ecb5f124e461" containerID="e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca" exitCode=0
Sep 09 21:19:30.662042 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:30.661654    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"
Sep 09 21:19:30.662042 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:30.661656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x" event={"ID":"2f850012-11d6-4b27-92e7-ecb5f124e461","Type":"ContainerDied","Data":"e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca"}
Sep 09 21:19:30.662042 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:30.661781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x" event={"ID":"2f850012-11d6-4b27-92e7-ecb5f124e461","Type":"ContainerDied","Data":"e5b82611c50a03e82936b2b2d26f733c9deca7d919b2a8a0e8aaffbafc377a35"}
Sep 09 21:19:30.662042 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:30.661811    2578 scope.go:117] "RemoveContainer" containerID="e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca"
Sep 09 21:19:30.670645 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:30.670629    2578 scope.go:117] "RemoveContainer" containerID="e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca"
Sep 09 21:19:30.670893 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:19:30.670878    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca\": container with ID starting with e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca not found: ID does not exist" containerID="e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca"
Sep 09 21:19:30.670939 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:30.670899    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca"} err="failed to get container status \"e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca\": rpc error: code = NotFound desc = could not find container \"e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca\": container with ID starting with e9d3a9ad4a61e2f5fc405c9929173e8404bca8de751837cb0953e420d3a0c2ca not found: ID does not exist"
Sep 09 21:19:30.682016 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:30.681993    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"]
Sep 09 21:19:30.684729 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:30.684709    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d7f624-5b64bddd9f-9dh5x"]
Sep 09 21:19:31.841051 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:31.841022    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f850012-11d6-4b27-92e7-ecb5f124e461" path="/var/lib/kubelet/pods/2f850012-11d6-4b27-92e7-ecb5f124e461/volumes"
Sep 09 21:19:40.107972 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:19:40.107935    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:20:19.409483 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:19.409447    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"]
Sep 09 21:20:19.410005 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:19.409666    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy" containerID="cri-o://f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343" gracePeriod=30
Sep 09 21:20:24.654588 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.654564    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:20:24.749029 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.748938    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-token\") pod \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") "
Sep 09 21:20:24.749029 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckkvf\" (UniqueName: \"kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-kube-api-access-ckkvf\") pod \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") "
Sep 09 21:20:24.749250 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749049    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-credential-socket\") pod \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") "
Sep 09 21:20:24.749250 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749095    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-podinfo\") pod \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") "
Sep 09 21:20:24.749250 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-data\") pod \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") "
Sep 09 21:20:24.749250 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749202    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-socket\") pod \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") "
Sep 09 21:20:24.749250 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749241    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istiod-ca-cert\") pod \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") "
Sep 09 21:20:24.749509 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749278    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-certs\") pod \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") "
Sep 09 21:20:24.749509 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749302    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-envoy\") pod \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\" (UID: \"8fc9af60-1868-4d0b-af7f-7d4efe9914a7\") "
Sep 09 21:20:24.749509 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749469    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8fc9af60-1868-4d0b-af7f-7d4efe9914a7" (UID: "8fc9af60-1868-4d0b-af7f-7d4efe9914a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:24.749509 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749480    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "8fc9af60-1868-4d0b-af7f-7d4efe9914a7" (UID: "8fc9af60-1868-4d0b-af7f-7d4efe9914a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:24.749716 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749521    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8fc9af60-1868-4d0b-af7f-7d4efe9914a7" (UID: "8fc9af60-1868-4d0b-af7f-7d4efe9914a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:24.749716 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749626    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.749716 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749674    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.749716 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749691    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.749891 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749712    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8fc9af60-1868-4d0b-af7f-7d4efe9914a7" (UID: "8fc9af60-1868-4d0b-af7f-7d4efe9914a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:24.749891 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.749781    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8fc9af60-1868-4d0b-af7f-7d4efe9914a7" (UID: "8fc9af60-1868-4d0b-af7f-7d4efe9914a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:24.751412 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.751384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8fc9af60-1868-4d0b-af7f-7d4efe9914a7" (UID: "8fc9af60-1868-4d0b-af7f-7d4efe9914a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:20:24.751571 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.751552    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8fc9af60-1868-4d0b-af7f-7d4efe9914a7" (UID: "8fc9af60-1868-4d0b-af7f-7d4efe9914a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:24.751676 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.751661    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-kube-api-access-ckkvf" (OuterVolumeSpecName: "kube-api-access-ckkvf") pod "8fc9af60-1868-4d0b-af7f-7d4efe9914a7" (UID: "8fc9af60-1868-4d0b-af7f-7d4efe9914a7"). InnerVolumeSpecName "kube-api-access-ckkvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:24.751715 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.751671    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "8fc9af60-1868-4d0b-af7f-7d4efe9914a7" (UID: "8fc9af60-1868-4d0b-af7f-7d4efe9914a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:24.850692 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.850660    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.850692 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.850688    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.850692 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.850698    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.850916 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.850706    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.850916 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.850714    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.850916 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.850722    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckkvf\" (UniqueName: \"kubernetes.io/projected/8fc9af60-1868-4d0b-af7f-7d4efe9914a7-kube-api-access-ckkvf\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.906049 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.906018    2578 generic.go:356] "Generic (PLEG): container finished" podID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerID="f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343" exitCode=0
Sep 09 21:20:24.906228 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.906067    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"
Sep 09 21:20:24.906228 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.906107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" event={"ID":"8fc9af60-1868-4d0b-af7f-7d4efe9914a7","Type":"ContainerDied","Data":"f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343"}
Sep 09 21:20:24.906228 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.906166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs" event={"ID":"8fc9af60-1868-4d0b-af7f-7d4efe9914a7","Type":"ContainerDied","Data":"28c877a3b272cd4af107bc796574693b4b71f7560717ebf34146ba236540c63a"}
Sep 09 21:20:24.906228 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.906207    2578 scope.go:117] "RemoveContainer" containerID="f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343"
Sep 09 21:20:24.915314 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.915298    2578 scope.go:117] "RemoveContainer" containerID="5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551"
Sep 09 21:20:24.924808 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.924784    2578 scope.go:117] "RemoveContainer" containerID="f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343"
Sep 09 21:20:24.925073 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:20:24.925051    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343\": container with ID starting with f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343 not found: ID does not exist" containerID="f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343"
Sep 09 21:20:24.925144 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.925086    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343"} err="failed to get container status \"f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343\": rpc error: code = NotFound desc = could not find container \"f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343\": container with ID starting with f8f49485ce0b6f2bf5e8957bc376742c20bfa940485416a4528000388b6af343 not found: ID does not exist"
Sep 09 21:20:24.925144 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.925110    2578 scope.go:117] "RemoveContainer" containerID="5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551"
Sep 09 21:20:24.925406 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:20:24.925389    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551\": container with ID starting with 5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551 not found: ID does not exist" containerID="5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551"
Sep 09 21:20:24.925460 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.925412    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551"} err="failed to get container status \"5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551\": rpc error: code = NotFound desc = could not find container \"5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551\": container with ID starting with 5cd37df6e85fd422bcd4f7c9338c77beea05fa4ca5d46daaa7fa9301d304f551 not found: ID does not exist"
Sep 09 21:20:24.930632 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.930610    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"]
Sep 09 21:20:24.935017 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:24.934996    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d7f624-openshift-default-7784b49d7b-xg4fs"]
Sep 09 21:20:25.842288 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:25.842258    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" path="/var/lib/kubelet/pods/8fc9af60-1868-4d0b-af7f-7d4efe9914a7/volumes"
Sep 09 21:20:30.409273 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.409240    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"]
Sep 09 21:20:30.409719 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.409530    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2" podUID="d3a2b388-a502-4d55-b484-171a60cae7d6" containerName="maas-api" containerID="cri-o://ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da" gracePeriod=30
Sep 09 21:20:30.650998 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.650968    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:20:30.799724 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.799643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlzh4\" (UniqueName: \"kubernetes.io/projected/d3a2b388-a502-4d55-b484-171a60cae7d6-kube-api-access-xlzh4\") pod \"d3a2b388-a502-4d55-b484-171a60cae7d6\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") "
Sep 09 21:20:30.799724 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.799698    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a2b388-a502-4d55-b484-171a60cae7d6-openshift-service-ca\") pod \"d3a2b388-a502-4d55-b484-171a60cae7d6\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") "
Sep 09 21:20:30.799724 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.799718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-metrics-certs\") pod \"d3a2b388-a502-4d55-b484-171a60cae7d6\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") "
Sep 09 21:20:30.799962 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.799742    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-maas-api-tls\") pod \"d3a2b388-a502-4d55-b484-171a60cae7d6\" (UID: \"d3a2b388-a502-4d55-b484-171a60cae7d6\") "
Sep 09 21:20:30.800099 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.800059    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a2b388-a502-4d55-b484-171a60cae7d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3a2b388-a502-4d55-b484-171a60cae7d6" (UID: "d3a2b388-a502-4d55-b484-171a60cae7d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:30.801905 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.801880    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a2b388-a502-4d55-b484-171a60cae7d6-kube-api-access-xlzh4" (OuterVolumeSpecName: "kube-api-access-xlzh4") pod "d3a2b388-a502-4d55-b484-171a60cae7d6" (UID: "d3a2b388-a502-4d55-b484-171a60cae7d6"). InnerVolumeSpecName "kube-api-access-xlzh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:30.801989 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.801903    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d3a2b388-a502-4d55-b484-171a60cae7d6" (UID: "d3a2b388-a502-4d55-b484-171a60cae7d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:20:30.801989 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.801921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3a2b388-a502-4d55-b484-171a60cae7d6" (UID: "d3a2b388-a502-4d55-b484-171a60cae7d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:20:30.901004 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.900975    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlzh4\" (UniqueName: \"kubernetes.io/projected/d3a2b388-a502-4d55-b484-171a60cae7d6-kube-api-access-xlzh4\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:30.901004 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.901001    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a2b388-a502-4d55-b484-171a60cae7d6-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:30.901004 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.901010    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:30.901266 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.901025    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a2b388-a502-4d55-b484-171a60cae7d6-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:20:30.933647 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.933616    2578 generic.go:356] "Generic (PLEG): container finished" podID="d3a2b388-a502-4d55-b484-171a60cae7d6" containerID="ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da" exitCode=0
Sep 09 21:20:30.933777 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.933648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2" event={"ID":"d3a2b388-a502-4d55-b484-171a60cae7d6","Type":"ContainerDied","Data":"ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da"}
Sep 09 21:20:30.933777 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.933668    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"
Sep 09 21:20:30.933777 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.933705    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2" event={"ID":"d3a2b388-a502-4d55-b484-171a60cae7d6","Type":"ContainerDied","Data":"9f22c72cb37db70adbb49577d4aeed624ef6304dc49d7e55fd12f7b449a94c68"}
Sep 09 21:20:30.933777 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.933728    2578 scope.go:117] "RemoveContainer" containerID="ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da"
Sep 09 21:20:30.942796 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.942780    2578 scope.go:117] "RemoveContainer" containerID="ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da"
Sep 09 21:20:30.943031 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:20:30.943016    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da\": container with ID starting with ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da not found: ID does not exist" containerID="ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da"
Sep 09 21:20:30.943082 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.943037    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da"} err="failed to get container status \"ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da\": rpc error: code = NotFound desc = could not find container \"ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da\": container with ID starting with ff6b5903a7e24da51120cee60dedf774574582b5d557ddffec0b3f4e44cb22da not found: ID does not exist"
Sep 09 21:20:30.960142 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.960119    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"]
Sep 09 21:20:30.966476 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:30.966456    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7dd56-cc9d9cccc-rhnd2"]
Sep 09 21:20:31.840298 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:31.840264    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3a2b388-a502-4d55-b484-171a60cae7d6" path="/var/lib/kubelet/pods/d3a2b388-a502-4d55-b484-171a60cae7d6/volumes"
Sep 09 21:20:33.824438 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:33.824407    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:20:51.632976 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:20:51.632925    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:21:07.685870 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:07.685839    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"]
Sep 09 21:21:07.686353 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:07.686037    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6" podUID="3ea85657-996d-4ca4-98fd-a01edc768074" containerName="maas-api" containerID="cri-o://a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3" gracePeriod=30
Sep 09 21:21:07.930678 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:07.930655    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:21:07.997704 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:07.997623    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea85657-996d-4ca4-98fd-a01edc768074-openshift-service-ca\") pod \"3ea85657-996d-4ca4-98fd-a01edc768074\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") "
Sep 09 21:21:07.997853 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:07.997721    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-metrics-certs\") pod \"3ea85657-996d-4ca4-98fd-a01edc768074\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") "
Sep 09 21:21:07.997853 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:07.997754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-maas-api-tls\") pod \"3ea85657-996d-4ca4-98fd-a01edc768074\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") "
Sep 09 21:21:07.997853 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:07.997804    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/3ea85657-996d-4ca4-98fd-a01edc768074-kube-api-access-5th4k\") pod \"3ea85657-996d-4ca4-98fd-a01edc768074\" (UID: \"3ea85657-996d-4ca4-98fd-a01edc768074\") "
Sep 09 21:21:07.998033 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:07.997995    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea85657-996d-4ca4-98fd-a01edc768074-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ea85657-996d-4ca4-98fd-a01edc768074" (UID: "3ea85657-996d-4ca4-98fd-a01edc768074"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:21:07.999988 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:07.999959    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ea85657-996d-4ca4-98fd-a01edc768074" (UID: "3ea85657-996d-4ca4-98fd-a01edc768074"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:21:08.000117 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:07.999976    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea85657-996d-4ca4-98fd-a01edc768074-kube-api-access-5th4k" (OuterVolumeSpecName: "kube-api-access-5th4k") pod "3ea85657-996d-4ca4-98fd-a01edc768074" (UID: "3ea85657-996d-4ca4-98fd-a01edc768074"). InnerVolumeSpecName "kube-api-access-5th4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:21:08.000242 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.000217    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3ea85657-996d-4ca4-98fd-a01edc768074" (UID: "3ea85657-996d-4ca4-98fd-a01edc768074"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:21:08.089248 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.089215    2578 generic.go:356] "Generic (PLEG): container finished" podID="3ea85657-996d-4ca4-98fd-a01edc768074" containerID="a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3" exitCode=0
Sep 09 21:21:08.089419 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.089261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6" event={"ID":"3ea85657-996d-4ca4-98fd-a01edc768074","Type":"ContainerDied","Data":"a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3"}
Sep 09 21:21:08.089419 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.089267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"
Sep 09 21:21:08.089419 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.089330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6" event={"ID":"3ea85657-996d-4ca4-98fd-a01edc768074","Type":"ContainerDied","Data":"3c3fe454af47688d81d40a657dc5daf56d80d01b9a1ed56ba69ec22c1039c1de"}
Sep 09 21:21:08.089419 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.089351    2578 scope.go:117] "RemoveContainer" containerID="a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3"
Sep 09 21:21:08.097912 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.097897    2578 scope.go:117] "RemoveContainer" containerID="a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3"
Sep 09 21:21:08.098204 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:21:08.098157    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3\": container with ID starting with a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3 not found: ID does not exist" containerID="a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3"
Sep 09 21:21:08.098306 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.098216    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3"} err="failed to get container status \"a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3\": rpc error: code = NotFound desc = could not find container \"a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3\": container with ID starting with a738b5371a7421d9d10dd49dbe4c07aaf152bb9976679632fb326cd3b3726ac3 not found: ID does not exist"
Sep 09 21:21:08.098375 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.098356    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea85657-996d-4ca4-98fd-a01edc768074-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:08.098422 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.098380    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:08.098422 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.098389    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ea85657-996d-4ca4-98fd-a01edc768074-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:08.098422 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.098397    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/3ea85657-996d-4ca4-98fd-a01edc768074-kube-api-access-5th4k\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:08.122745 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.122715    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"]
Sep 09 21:21:08.129973 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:08.129948    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8db5a7-7cb7b7b695-9txq6"]
Sep 09 21:21:09.840764 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:09.840735    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea85657-996d-4ca4-98fd-a01edc768074" path="/var/lib/kubelet/pods/3ea85657-996d-4ca4-98fd-a01edc768074/volumes"
Sep 09 21:21:10.518282 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:10.518251    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:21:15.419823 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:15.419779    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"]
Sep 09 21:21:15.420238 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:15.420014    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy" containerID="cri-o://2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5" gracePeriod=30
Sep 09 21:21:18.531009 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:18.530968    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 21:21:18.531395 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:18.531051    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:21:19.721854 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:19.721818    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:21:20.656593 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.656570    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:21:20.812009 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.811929    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-podinfo\") pod \"22feda42-4b1e-40b5-8541-71ae2b1309c0\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") "
Sep 09 21:21:20.812009 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.811964    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-certs\") pod \"22feda42-4b1e-40b5-8541-71ae2b1309c0\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") "
Sep 09 21:21:20.812009 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.811997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxtpb\" (UniqueName: \"kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-kube-api-access-qxtpb\") pod \"22feda42-4b1e-40b5-8541-71ae2b1309c0\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") "
Sep 09 21:21:20.812590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-credential-socket\") pod \"22feda42-4b1e-40b5-8541-71ae2b1309c0\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") "
Sep 09 21:21:20.812590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812038    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-data\") pod \"22feda42-4b1e-40b5-8541-71ae2b1309c0\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") "
Sep 09 21:21:20.812590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812063    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-token\") pod \"22feda42-4b1e-40b5-8541-71ae2b1309c0\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") "
Sep 09 21:21:20.812590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-socket\") pod \"22feda42-4b1e-40b5-8541-71ae2b1309c0\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") "
Sep 09 21:21:20.812590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812117    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22feda42-4b1e-40b5-8541-71ae2b1309c0-istiod-ca-cert\") pod \"22feda42-4b1e-40b5-8541-71ae2b1309c0\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") "
Sep 09 21:21:20.812590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812196    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-envoy\") pod \"22feda42-4b1e-40b5-8541-71ae2b1309c0\" (UID: \"22feda42-4b1e-40b5-8541-71ae2b1309c0\") "
Sep 09 21:21:20.812590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812224    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22feda42-4b1e-40b5-8541-71ae2b1309c0" (UID: "22feda42-4b1e-40b5-8541-71ae2b1309c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:20.812590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812428    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:20.812590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812422    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "22feda42-4b1e-40b5-8541-71ae2b1309c0" (UID: "22feda42-4b1e-40b5-8541-71ae2b1309c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:20.812590 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22feda42-4b1e-40b5-8541-71ae2b1309c0" (UID: "22feda42-4b1e-40b5-8541-71ae2b1309c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:20.813131 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22feda42-4b1e-40b5-8541-71ae2b1309c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22feda42-4b1e-40b5-8541-71ae2b1309c0" (UID: "22feda42-4b1e-40b5-8541-71ae2b1309c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:21:20.813131 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.812668    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22feda42-4b1e-40b5-8541-71ae2b1309c0" (UID: "22feda42-4b1e-40b5-8541-71ae2b1309c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:20.814414 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.814389    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22feda42-4b1e-40b5-8541-71ae2b1309c0" (UID: "22feda42-4b1e-40b5-8541-71ae2b1309c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:21:20.814414 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.814398    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-kube-api-access-qxtpb" (OuterVolumeSpecName: "kube-api-access-qxtpb") pod "22feda42-4b1e-40b5-8541-71ae2b1309c0" (UID: "22feda42-4b1e-40b5-8541-71ae2b1309c0"). InnerVolumeSpecName "kube-api-access-qxtpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:21:20.814549 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.814439    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "22feda42-4b1e-40b5-8541-71ae2b1309c0" (UID: "22feda42-4b1e-40b5-8541-71ae2b1309c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:21:20.814549 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.814452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22feda42-4b1e-40b5-8541-71ae2b1309c0" (UID: "22feda42-4b1e-40b5-8541-71ae2b1309c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:20.913708 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.913677    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxtpb\" (UniqueName: \"kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-kube-api-access-qxtpb\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:20.913708 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.913702    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:20.913897 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.913717    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:20.913897 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.913730    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:20.913897 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.913742    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:20.913897 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.913753    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22feda42-4b1e-40b5-8541-71ae2b1309c0-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:20.913897 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.913764    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:20.913897 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:20.913775    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22feda42-4b1e-40b5-8541-71ae2b1309c0-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:21:21.144083 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.144050    2578 generic.go:356] "Generic (PLEG): container finished" podID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerID="2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5" exitCode=0
Sep 09 21:21:21.144305 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.144103    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"
Sep 09 21:21:21.144305 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.144143    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" event={"ID":"22feda42-4b1e-40b5-8541-71ae2b1309c0","Type":"ContainerDied","Data":"2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5"}
Sep 09 21:21:21.144305 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.144231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd" event={"ID":"22feda42-4b1e-40b5-8541-71ae2b1309c0","Type":"ContainerDied","Data":"81ba6bcc260530007fff35018dbc17578d119e37bbffe83f440bc76d23a26820"}
Sep 09 21:21:21.144305 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.144262    2578 scope.go:117] "RemoveContainer" containerID="2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5"
Sep 09 21:21:21.152909 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.152889    2578 scope.go:117] "RemoveContainer" containerID="610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a"
Sep 09 21:21:21.159971 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.159954    2578 scope.go:117] "RemoveContainer" containerID="2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5"
Sep 09 21:21:21.160230 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:21:21.160209    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5\": container with ID starting with 2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5 not found: ID does not exist" containerID="2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5"
Sep 09 21:21:21.160290 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.160239    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5"} err="failed to get container status \"2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5\": rpc error: code = NotFound desc = could not find container \"2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5\": container with ID starting with 2d6119c01a3d6f00a6e36bc79b4321be2f5ca3e7c0c0cac2deeca9ed1ad032d5 not found: ID does not exist"
Sep 09 21:21:21.160290 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.160257    2578 scope.go:117] "RemoveContainer" containerID="610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a"
Sep 09 21:21:21.160457 ip-10-0-136-121 kubenswrapper[2578]: E0909 21:21:21.160442    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a\": container with ID starting with 610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a not found: ID does not exist" containerID="610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a"
Sep 09 21:21:21.160504 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.160459    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a"} err="failed to get container status \"610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a\": rpc error: code = NotFound desc = could not find container \"610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a\": container with ID starting with 610ea43788dbfc6d51a1da52c68809cf086b81c090ab98f9a41a500f9d0b250a not found: ID does not exist"
Sep 09 21:21:21.167815 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.167794    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"]
Sep 09 21:21:21.171132 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.171113    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8db5a7-openshift-default-6cd76448b4-lb2kd"]
Sep 09 21:21:21.844966 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:21.844930    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" path="/var/lib/kubelet/pods/22feda42-4b1e-40b5-8541-71ae2b1309c0/volumes"
Sep 09 21:21:44.044339 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:21:44.044309    2578 scope.go:117] "RemoveContainer" containerID="f24ec610ab1f87b5f079e9b1c9bc927b04e547a407f4e949ba002f8371daee28"
Sep 09 21:22:43.028059 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:22:43.028019    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:23:09.018950 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:23:09.018866    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:23:16.926098 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:23:16.926061    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:23:25.712872 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:23:25.712838    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:23:42.714555 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:23:42.714520    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:23:51.109992 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:23:51.109947    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:25:17.012903 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:25:17.012868    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:25:25.730204 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:25:25.730153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:25:35.173287 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:25:35.173254    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:25:52.840289 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:25:52.840254    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:26:11.410594 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:11.410553    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:26:20.509778 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:20.509744    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:26:29.704878 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:29.704841    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:26:30.112095 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112056    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-q7grs"]
Sep 09 21:26:30.112356 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112343    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a2b388-a502-4d55-b484-171a60cae7d6" containerName="maas-api"
Sep 09 21:26:30.112403 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112359    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a2b388-a502-4d55-b484-171a60cae7d6" containerName="maas-api"
Sep 09 21:26:30.112403 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112372    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103c73a4-78fd-455c-80be-1500534369ff" containerName="maas-api"
Sep 09 21:26:30.112403 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112377    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="103c73a4-78fd-455c-80be-1500534369ff" containerName="maas-api"
Sep 09 21:26:30.112403 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112387    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy"
Sep 09 21:26:30.112403 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112392    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy"
Sep 09 21:26:30.112403 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112401    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112408    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112419    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f850012-11d6-4b27-92e7-ecb5f124e461" containerName="maas-api"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112428    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f850012-11d6-4b27-92e7-ecb5f124e461" containerName="maas-api"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112438    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea85657-996d-4ca4-98fd-a01edc768074" containerName="maas-api"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112446    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea85657-996d-4ca4-98fd-a01edc768074" containerName="maas-api"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112453    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112458    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112466    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112471    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112527    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a2b388-a502-4d55-b484-171a60cae7d6" containerName="maas-api"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112533    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112539    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112545    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22feda42-4b1e-40b5-8541-71ae2b1309c0" containerName="istio-proxy"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112550    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f850012-11d6-4b27-92e7-ecb5f124e461" containerName="maas-api"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112557    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="103c73a4-78fd-455c-80be-1500534369ff" containerName="maas-api"
Sep 09 21:26:30.112589 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.112564    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea85657-996d-4ca4-98fd-a01edc768074" containerName="maas-api"
Sep 09 21:26:30.115409 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.115393    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.123264 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.123161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-gateway-identity\""
Sep 09 21:26:30.123436 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.123270    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gj4rp\""
Sep 09 21:26:30.123436 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.123282    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 21:26:30.123436 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.123273    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:26:30.123436 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.123209    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 21:26:30.123436 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.123221    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 21:26:30.128315 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.128290    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-q7grs"]
Sep 09 21:26:30.204113 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.204076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d7a25a-ee57-47d8-9a2c-1d046719e77b-openshift-service-ca\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.204318 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.204121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dfx2\" (UniqueName: \"kubernetes.io/projected/30d7a25a-ee57-47d8-9a2c-1d046719e77b-kube-api-access-2dfx2\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.204318 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.204245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-maas-api-tls\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.204318 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.204292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-metrics-certs\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.305060 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.305009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-maas-api-tls\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.305060 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.305052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-metrics-certs\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.305358 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.305079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d7a25a-ee57-47d8-9a2c-1d046719e77b-openshift-service-ca\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.305358 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.305110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dfx2\" (UniqueName: \"kubernetes.io/projected/30d7a25a-ee57-47d8-9a2c-1d046719e77b-kube-api-access-2dfx2\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.305800 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.305774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d7a25a-ee57-47d8-9a2c-1d046719e77b-openshift-service-ca\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.307575 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.307552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-metrics-certs\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.307656 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.307639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-maas-api-tls\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.313871 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.313845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dfx2\" (UniqueName: \"kubernetes.io/projected/30d7a25a-ee57-47d8-9a2c-1d046719e77b-kube-api-access-2dfx2\") pod \"maas-api-68d778f44-q7grs\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.431599 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.431523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:30.703064 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.703036    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-q7grs"]
Sep 09 21:26:30.705495 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:26:30.705466    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d7a25a_ee57_47d8_9a2c_1d046719e77b.slice/crio-3ce8ee3558b6fe64b2f8792a4f72a09e6f1bca38144e1720cd78acec10311540 WatchSource:0}: Error finding container 3ce8ee3558b6fe64b2f8792a4f72a09e6f1bca38144e1720cd78acec10311540: Status 404 returned error can't find the container with id 3ce8ee3558b6fe64b2f8792a4f72a09e6f1bca38144e1720cd78acec10311540
Sep 09 21:26:30.706784 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:30.706761    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:26:31.371419 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:31.371386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs" event={"ID":"30d7a25a-ee57-47d8-9a2c-1d046719e77b","Type":"ContainerStarted","Data":"169f09f64078bfc9da66f02f717f2e1b5e53ad484fe875dda5b2449398bfdead"}
Sep 09 21:26:31.371592 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:31.371438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs" event={"ID":"30d7a25a-ee57-47d8-9a2c-1d046719e77b","Type":"ContainerStarted","Data":"3ce8ee3558b6fe64b2f8792a4f72a09e6f1bca38144e1720cd78acec10311540"}
Sep 09 21:26:31.371592 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:31.371494    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:31.390156 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:31.390033    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs" podStartSLOduration=0.971571108 podStartE2EDuration="1.390016146s" totalImagesPullingTime="418.445038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:26:31.389426336 +0000 UTC m=+2268.044206494" watchObservedRunningTime="2026-09-09 21:26:31.390016146 +0000 UTC m=+2268.044796304"
Sep 09 21:26:37.272546 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.272515    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-q7grs"]
Sep 09 21:26:37.272981 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.272697    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs" podUID="30d7a25a-ee57-47d8-9a2c-1d046719e77b" containerName="maas-api" containerID="cri-o://169f09f64078bfc9da66f02f717f2e1b5e53ad484fe875dda5b2449398bfdead" gracePeriod=30
Sep 09 21:26:37.277947 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.277921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:37.316997 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.316964    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-584858f848-n96v5"]
Sep 09 21:26:37.317388 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.317370    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc9af60-1868-4d0b-af7f-7d4efe9914a7" containerName="istio-proxy"
Sep 09 21:26:37.320399 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.320378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.336583 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.336561    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-584858f848-n96v5"]
Sep 09 21:26:37.397104 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.397079    2578 generic.go:356] "Generic (PLEG): container finished" podID="30d7a25a-ee57-47d8-9a2c-1d046719e77b" containerID="169f09f64078bfc9da66f02f717f2e1b5e53ad484fe875dda5b2449398bfdead" exitCode=0
Sep 09 21:26:37.397230 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.397166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs" event={"ID":"30d7a25a-ee57-47d8-9a2c-1d046719e77b","Type":"ContainerDied","Data":"169f09f64078bfc9da66f02f717f2e1b5e53ad484fe875dda5b2449398bfdead"}
Sep 09 21:26:37.465658 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.465629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff043f8-ac0a-4c06-b752-9133935c84fe-maas-api-tls\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.465807 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.465667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff043f8-ac0a-4c06-b752-9133935c84fe-metrics-certs\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.465807 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.465684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2v6\" (UniqueName: \"kubernetes.io/projected/fff043f8-ac0a-4c06-b752-9133935c84fe-kube-api-access-7t2v6\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.465807 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.465787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff043f8-ac0a-4c06-b752-9133935c84fe-openshift-service-ca\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.503285 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.503261    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:37.508554 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.508533    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:26:37.566868 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.566795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff043f8-ac0a-4c06-b752-9133935c84fe-maas-api-tls\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.566868 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.566834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff043f8-ac0a-4c06-b752-9133935c84fe-metrics-certs\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.566868 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.566859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2v6\" (UniqueName: \"kubernetes.io/projected/fff043f8-ac0a-4c06-b752-9133935c84fe-kube-api-access-7t2v6\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.567098 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.566981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff043f8-ac0a-4c06-b752-9133935c84fe-openshift-service-ca\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.567608 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.567584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff043f8-ac0a-4c06-b752-9133935c84fe-openshift-service-ca\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.569267 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.569242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff043f8-ac0a-4c06-b752-9133935c84fe-metrics-certs\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.569350 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.569302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff043f8-ac0a-4c06-b752-9133935c84fe-maas-api-tls\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.574824 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.574803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2v6\" (UniqueName: \"kubernetes.io/projected/fff043f8-ac0a-4c06-b752-9133935c84fe-kube-api-access-7t2v6\") pod \"maas-api-584858f848-n96v5\" (UID: \"fff043f8-ac0a-4c06-b752-9133935c84fe\") " pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.656327 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.656297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:37.668133 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.668109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d7a25a-ee57-47d8-9a2c-1d046719e77b-openshift-service-ca\") pod \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") "
Sep 09 21:26:37.668256 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.668152    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-metrics-certs\") pod \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") "
Sep 09 21:26:37.668256 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.668213    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dfx2\" (UniqueName: \"kubernetes.io/projected/30d7a25a-ee57-47d8-9a2c-1d046719e77b-kube-api-access-2dfx2\") pod \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") "
Sep 09 21:26:37.668381 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.668281    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-maas-api-tls\") pod \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\" (UID: \"30d7a25a-ee57-47d8-9a2c-1d046719e77b\") "
Sep 09 21:26:37.668480 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.668454    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d7a25a-ee57-47d8-9a2c-1d046719e77b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30d7a25a-ee57-47d8-9a2c-1d046719e77b" (UID: "30d7a25a-ee57-47d8-9a2c-1d046719e77b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:26:37.668640 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.668599    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d7a25a-ee57-47d8-9a2c-1d046719e77b-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:26:37.670438 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.670412    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "30d7a25a-ee57-47d8-9a2c-1d046719e77b" (UID: "30d7a25a-ee57-47d8-9a2c-1d046719e77b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:26:37.670520 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.670451    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d7a25a-ee57-47d8-9a2c-1d046719e77b-kube-api-access-2dfx2" (OuterVolumeSpecName: "kube-api-access-2dfx2") pod "30d7a25a-ee57-47d8-9a2c-1d046719e77b" (UID: "30d7a25a-ee57-47d8-9a2c-1d046719e77b"). InnerVolumeSpecName "kube-api-access-2dfx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:26:37.670582 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.670567    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30d7a25a-ee57-47d8-9a2c-1d046719e77b" (UID: "30d7a25a-ee57-47d8-9a2c-1d046719e77b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:26:37.769938 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.769906    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:26:37.769938 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.769935    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d7a25a-ee57-47d8-9a2c-1d046719e77b-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:26:37.770107 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.769947    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dfx2\" (UniqueName: \"kubernetes.io/projected/30d7a25a-ee57-47d8-9a2c-1d046719e77b-kube-api-access-2dfx2\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 09 21:26:37.930208 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:37.930162    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-584858f848-n96v5"]
Sep 09 21:26:37.933006 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:26:37.932969    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff043f8_ac0a_4c06_b752_9133935c84fe.slice/crio-d183c935bc2dd32afc2e1ac7471261e5f23b47406b86b6a5a05825f3db90495d WatchSource:0}: Error finding container d183c935bc2dd32afc2e1ac7471261e5f23b47406b86b6a5a05825f3db90495d: Status 404 returned error can't find the container with id d183c935bc2dd32afc2e1ac7471261e5f23b47406b86b6a5a05825f3db90495d
Sep 09 21:26:38.402529 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:38.402500    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs"
Sep 09 21:26:38.402529 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:38.402508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68d778f44-q7grs" event={"ID":"30d7a25a-ee57-47d8-9a2c-1d046719e77b","Type":"ContainerDied","Data":"3ce8ee3558b6fe64b2f8792a4f72a09e6f1bca38144e1720cd78acec10311540"}
Sep 09 21:26:38.403024 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:38.402576    2578 scope.go:117] "RemoveContainer" containerID="169f09f64078bfc9da66f02f717f2e1b5e53ad484fe875dda5b2449398bfdead"
Sep 09 21:26:38.403830 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:38.403754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5" event={"ID":"fff043f8-ac0a-4c06-b752-9133935c84fe","Type":"ContainerStarted","Data":"d183c935bc2dd32afc2e1ac7471261e5f23b47406b86b6a5a05825f3db90495d"}
Sep 09 21:26:38.425287 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:38.425266    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-q7grs"]
Sep 09 21:26:38.429025 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:38.429002    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-q7grs"]
Sep 09 21:26:39.410086 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:39.410053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5" event={"ID":"fff043f8-ac0a-4c06-b752-9133935c84fe","Type":"ContainerStarted","Data":"3745f61ce81917c1eabc3c7d4c987d70d9b78f98ab452172ffb449a3fb8b4f9b"}
Sep 09 21:26:39.410495 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:39.410185    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:39.430544 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:39.430501    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5" podStartSLOduration=1.950421677 podStartE2EDuration="2.430487s" totalImagesPullingTime="480.065323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:26:39.42839137 +0000 UTC m=+2276.083171541" watchObservedRunningTime="2026-09-09 21:26:39.430487 +0000 UTC m=+2276.085267158"
Sep 09 21:26:39.840884 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:39.840855    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30d7a25a-ee57-47d8-9a2c-1d046719e77b" path="/var/lib/kubelet/pods/30d7a25a-ee57-47d8-9a2c-1d046719e77b/volumes"
Sep 09 21:26:44.312335 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:44.312308    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:26:45.419586 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:45.419555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-584858f848-n96v5"
Sep 09 21:26:50.147816 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:50.147779    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:26:57.311772 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:26:57.311735    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tffg"]
Sep 09 21:27:05.689928 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:05.689902    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d75fb655b-w4s89_eec765b4-c74c-4756-9bdb-f5720d78b735/copy-manifests/0.log"
Sep 09 21:27:05.696716 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:05.696697    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d75fb655b-w4s89_eec765b4-c74c-4756-9bdb-f5720d78b735/manager/0.log"
Sep 09 21:27:06.134947 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:06.134916    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-584858f848-n96v5_fff043f8-ac0a-4c06-b752-9133935c84fe/maas-api/0.log"
Sep 09 21:27:07.693278 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:07.693247    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6tffg_3cdd3e1b-8518-42ed-a96a-19501b6602bf/limitador/0.log"
Sep 09 21:27:08.268796 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:08.268755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vrdgd_42ac86f4-2be9-4205-87b0-f19c31e524f7/discovery/0.log"
Sep 09 21:27:09.753100 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:09.753016    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpcmbq_1e54b784-eb60-41ee-aa65-cb7b7b2aaad8/main/0.log"
Sep 09 21:27:09.858495 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:09.858463    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-7bh5n_509facc5-eeea-41cf-b580-00a0f69763a1/main/0.log"
Sep 09 21:27:18.346343 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:18.346288    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2wnz7_d44783b3-1045-4e33-83de-7380e2678e62/global-pull-secret-syncer/0.log"
Sep 09 21:27:18.526134 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:18.526107    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d8rzb_0282f2d8-4948-4297-94a6-9da942c5a7be/konnectivity-agent/0.log"
Sep 09 21:27:18.654774 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:18.654687    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-121.ec2.internal_1a618664de1a81cb8ba66310981e215f/haproxy/0.log"
Sep 09 21:27:23.569509 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:23.569478    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6tffg_3cdd3e1b-8518-42ed-a96a-19501b6602bf/limitador/0.log"
Sep 09 21:27:25.465580 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:25.465542    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gjnj2_be4bf9d0-2fc0-4354-8380-81a52e719a5d/node-exporter/0.log"
Sep 09 21:27:25.489405 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:25.489380    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gjnj2_be4bf9d0-2fc0-4354-8380-81a52e719a5d/kube-rbac-proxy/0.log"
Sep 09 21:27:25.521325 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:25.521290    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gjnj2_be4bf9d0-2fc0-4354-8380-81a52e719a5d/init-textfile/0.log"
Sep 09 21:27:26.110556 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:26.110513    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-8kz5l_f24a2e20-1e63-4ff3-b2d5-6880648c43bd/prometheus-operator/0.log"
Sep 09 21:27:26.135310 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:26.135280    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-8kz5l_f24a2e20-1e63-4ff3-b2d5-6880648c43bd/kube-rbac-proxy/0.log"
Sep 09 21:27:26.161052 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:26.161033    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-wns8g_b662ffa4-442d-47e3-86ed-3359fe0b7b03/prometheus-operator-admission-webhook/0.log"
Sep 09 21:27:28.648354 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.648317    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"]
Sep 09 21:27:28.648819 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.648735    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d7a25a-ee57-47d8-9a2c-1d046719e77b" containerName="maas-api"
Sep 09 21:27:28.648819 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.648753    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d7a25a-ee57-47d8-9a2c-1d046719e77b" containerName="maas-api"
Sep 09 21:27:28.648955 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.648831    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d7a25a-ee57-47d8-9a2c-1d046719e77b" containerName="maas-api"
Sep 09 21:27:28.651819 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.651797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.654554 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.654538    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7p779\"/\"default-dockercfg-fxqvc\""
Sep 09 21:27:28.654667 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.654568    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7p779\"/\"kube-root-ca.crt\""
Sep 09 21:27:28.654759 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.654744    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7p779\"/\"openshift-service-ca.crt\""
Sep 09 21:27:28.663492 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.663473    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"]
Sep 09 21:27:28.767864 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.767826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-podres\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.768020 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.767915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-sys\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.768020 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.767969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-proc\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.768020 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.767991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpll7\" (UniqueName: \"kubernetes.io/projected/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-kube-api-access-rpll7\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.768020 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.768011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-lib-modules\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.868641 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.868606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-podres\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.868832 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.868658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-sys\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.868832 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.868728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-sys\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.868832 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.868757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-proc\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.868832 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.868776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpll7\" (UniqueName: \"kubernetes.io/projected/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-kube-api-access-rpll7\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.868832 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.868793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-lib-modules\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.868832 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.868803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-podres\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.869165 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.868869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-proc\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.869165 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.869099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-lib-modules\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.879927 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.879905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpll7\" (UniqueName: \"kubernetes.io/projected/32e3beb6-a09e-4e2f-986f-ddbe09c2819f-kube-api-access-rpll7\") pod \"perf-node-gather-daemonset-wvkkp\" (UID: \"32e3beb6-a09e-4e2f-986f-ddbe09c2819f\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:28.960987 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:28.960924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:29.228771 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:29.228748    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"]
Sep 09 21:27:29.231368 ip-10-0-136-121 kubenswrapper[2578]: W0909 21:27:29.231344    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod32e3beb6_a09e_4e2f_986f_ddbe09c2819f.slice/crio-6de7cd79e84ea27c043aacd78706587b9dc05ac56847fc220d76f277db022574 WatchSource:0}: Error finding container 6de7cd79e84ea27c043aacd78706587b9dc05ac56847fc220d76f277db022574: Status 404 returned error can't find the container with id 6de7cd79e84ea27c043aacd78706587b9dc05ac56847fc220d76f277db022574
Sep 09 21:27:29.617471 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:29.617435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp" event={"ID":"32e3beb6-a09e-4e2f-986f-ddbe09c2819f","Type":"ContainerStarted","Data":"ca5b39be3aabed9bfe57a447323dd6178306388e4a7f449a0e9584a18c7aba44"}
Sep 09 21:27:29.617660 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:29.617497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp" event={"ID":"32e3beb6-a09e-4e2f-986f-ddbe09c2819f","Type":"ContainerStarted","Data":"6de7cd79e84ea27c043aacd78706587b9dc05ac56847fc220d76f277db022574"}
Sep 09 21:27:29.617660 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:29.617621    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:31.158882 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:31.158849    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nglhm_8c27b357-bfd5-4cc0-8da1-8245185755bf/dns/0.log"
Sep 09 21:27:31.181515 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:31.181489    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nglhm_8c27b357-bfd5-4cc0-8da1-8245185755bf/kube-rbac-proxy/0.log"
Sep 09 21:27:31.280707 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:31.280674    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jfdfz_7fd38230-7de5-4519-bf5d-db7c7ea09cc2/dns-node-resolver/0.log"
Sep 09 21:27:31.909162 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:31.909135    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7tx4f_5ca402f4-5c17-48e7-bafd-2fe460254bcf/node-ca/0.log"
Sep 09 21:27:33.160776 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:33.160747    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vrdgd_42ac86f4-2be9-4205-87b0-f19c31e524f7/discovery/0.log"
Sep 09 21:27:34.227791 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:34.227762    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mtqlt_6155565e-bace-404d-8d02-daea0e9e697d/serve-healthcheck-canary/0.log"
Sep 09 21:27:34.896810 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:34.896782    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b5nrp_b022ae00-2834-4f4f-9f87-ba57d67c4702/kube-rbac-proxy/0.log"
Sep 09 21:27:34.919242 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:34.919214    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b5nrp_b022ae00-2834-4f4f-9f87-ba57d67c4702/exporter/0.log"
Sep 09 21:27:34.946102 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:34.946086    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b5nrp_b022ae00-2834-4f4f-9f87-ba57d67c4702/extractor/0.log"
Sep 09 21:27:35.631506 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:35.631481    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp"
Sep 09 21:27:35.650377 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:35.650333    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-wvkkp" podStartSLOduration=7.650319011 podStartE2EDuration="7.650319011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:27:29.634092932 +0000 UTC m=+2326.288873089" watchObservedRunningTime="2026-09-09 21:27:35.650319011 +0000 UTC m=+2332.305099206"
Sep 09 21:27:38.076870 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:38.076837    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d75fb655b-w4s89_eec765b4-c74c-4756-9bdb-f5720d78b735/manager/0.log"
Sep 09 21:27:38.103241 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:38.103220    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d75fb655b-w4s89_eec765b4-c74c-4756-9bdb-f5720d78b735/copy-manifests/0.log"
Sep 09 21:27:39.469971 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:39.469937    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-544fcc9ff7-m5lxx_769e7d4c-0867-428d-afe4-5e7faedd8efb/manager/0.log"
Sep 09 21:27:39.499460 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:39.499435    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-pzwsc_db961ba8-1738-4629-8345-eb9409b316d6/openshift-lws-operator/0.log"
Sep 09 21:27:39.545285 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:39.545254    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vrdgd_42ac86f4-2be9-4205-87b0-f19c31e524f7/discovery/0.log"
Sep 09 21:27:41.422819 ip-10-0-136-121 kubenswrapper[2578]: I0909 21:27:41.422757    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vrdgd_42ac86f4-2be9-4205-87b0-f19c31e524f7/discovery/0.log"