Sep 10 21:17:16.131416 ip-10-0-135-226 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 21:17:16.131426 ip-10-0-135-226 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 21:17:16.131433 ip-10-0-135-226 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 21:17:16.131581 ip-10-0-135-226 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 21:17:26.228925 ip-10-0-135-226 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 21:17:26.228945 ip-10-0-135-226 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 190240359d6743f4b0cf0d2cb70461b5 --
Sep 10 21:19:19.956595 ip-10-0-135-226 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 21:19:20.337650 ip-10-0-135-226 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 21:19:20.337650 ip-10-0-135-226 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 21:19:20.337650 ip-10-0-135-226 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 21:19:20.337650 ip-10-0-135-226 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 21:19:20.337650 ip-10-0-135-226 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 21:19:20.339080 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.338994    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 21:19:20.342623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342607    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 21:19:20.342623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342623    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342627    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342630    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342634    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342636    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342640    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342643    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342646    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342649    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342651    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342654    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342657    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342659    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342662    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342664    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342667    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342669    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342671    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342674    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342677    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 21:19:20.342684 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342680    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342687    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342690    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342693    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342696    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342699    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342702    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342704    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342707    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342709    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342712    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342714    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342717    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342719    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342722    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342725    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342728    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342730    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342734    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342738    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 21:19:20.343171 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342741    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342745    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342748    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342750    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342754    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342756    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342759    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342762    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342765    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342770    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342772    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342775    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342779    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342781    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342784    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342787    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342789    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342793    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342797    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 21:19:20.343704 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342800    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342802    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342805    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342808    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342811    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342813    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342816    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342818    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342821    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342823    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342826    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342828    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342831    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342833    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342836    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342839    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342841    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342844    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342846    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342848    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 21:19:20.344164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342851    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342854    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342856    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342858    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342861    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342863    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342866    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342869    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342873    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342876    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342878    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342881    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342884    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342886    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342889    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342891    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342894    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342896    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.342899    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343582    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 21:19:20.344660 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343589    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343592    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343595    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343600    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343603    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343607    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343609    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343613    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343617    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343620    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343622    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343625    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343630    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343634    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343637    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343640    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343642    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343645    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343648    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 21:19:20.345142 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343651    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343654    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343657    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343659    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343662    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343665    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343668    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343671    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343673    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343676    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343678    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343681    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343684    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343686    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343688    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343692    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343694    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343697    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343699    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343702    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 21:19:20.345637 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343704    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343707    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343711    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343714    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343716    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343719    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343721    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343724    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343726    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343729    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343731    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343734    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343737    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343739    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343742    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343745    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343747    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343750    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343752    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343755    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 21:19:20.346117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343758    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343760    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343762    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343765    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343767    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343770    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343772    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343774    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343777    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343779    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343782    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343784    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343787    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343789    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343792    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343795    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343798    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343800    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343803    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343805    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 21:19:20.346611 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343808    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343810    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343813    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343815    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343818    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343821    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343824    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343827    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343829    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343832    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343834    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343837    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343839    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343842    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343844    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343847    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343851    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343854    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.343856    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 21:19:20.347208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344208    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344213    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344216    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344218    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344221    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344224    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344227    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344230    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344233    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344236    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344239    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344242    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344244    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344247    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344249    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344252    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344254    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344257    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344260    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344262    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 21:19:20.347681 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344265    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344267    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344270    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344272    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344275    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344277    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344280    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344301    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344383    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344387    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344390    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344393    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344397    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344399    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344402    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344406    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344410    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344414    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344417    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 21:19:20.348164 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344420    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344423    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344428    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344431    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344435    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344437    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344440    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344443    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344446    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344450    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344454    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344457    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344460    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344462    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344465    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344468    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344471    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344474    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344476    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 21:19:20.348674 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344479    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344482    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344484    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344487    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344489    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344492    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344494    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344496    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344499    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344502    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344504    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344507    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344510    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344512    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344514    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344517    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344520    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344522    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344525    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344527    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 21:19:20.349133 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344530    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344533    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344536    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344540    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344542    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344545    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344547    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344550    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344553    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344555    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344558    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344560    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344562    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344565    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344568    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344570    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344572    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344575    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344577    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344580    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 21:19:20.349640 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.344582    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344659    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344694    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344704    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344709    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344713    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344716    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344720    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344724    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344728    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344731    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344734    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344737    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344740    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344743    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344746    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344749    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344752    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344754    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344758    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344761    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344764    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344767    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344770    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 21:19:20.350118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344774    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344776    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344779    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344782    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344785    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344788    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344791    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344794    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344797    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344801    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344803    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344808    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344811    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344814    2580 flags.go:64] FLAG: --enable-server="true"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344816    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344821    2580 flags.go:64] FLAG: --event-burst="100"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344824    2580 flags.go:64] FLAG: --event-qps="50"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344827    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344831    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344834    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344838    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344841    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344843    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344846    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344849    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 21:19:20.350689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344852    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344855    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344858    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344861    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344864    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344867    2580 flags.go:64] FLAG: --feature-gates=""
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344870    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344873    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344876    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344879    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344882    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344885    2580 flags.go:64] FLAG: --help="false"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344887    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344890    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344894    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344896    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344900    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344903    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344907    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344910    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344913    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344916    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344918    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344921    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 21:19:20.351328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344924    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344927    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344930    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344933    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344936    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344938    2580 flags.go:64] FLAG: --lock-file=""
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344941    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344944    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344947    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344953    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344955    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344958    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344961    2580 flags.go:64] FLAG: --logging-format="text"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344963    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344966    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344969    2580 flags.go:64] FLAG: --manifest-url=""
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344971    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344975    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344978    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344982    2580 flags.go:64] FLAG: --max-pods="110"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344985    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344988    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344991    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344993    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344996    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 21:19:20.351959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.344999    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345003    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345010    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345013    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345016    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345019    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345022    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345027    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345030    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345033    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345036    2580 flags.go:64] FLAG: --port="10250"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345040    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345043    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0da12a20ebbb069be"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345046    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345049    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345052    2580 flags.go:64] FLAG: --register-node="true"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345055    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345058    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345061    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345063    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345066    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345070    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345072    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345075    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345078    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 21:19:20.352567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345080    2580 flags.go:64] FLAG: --runonce="false"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345083    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345086    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345089    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345091    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345094    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345097    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345100    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345104    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345106    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345109    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345112    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345115    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345118    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345121    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345123    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345128    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345131    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345134    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345138    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345141    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345147    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345150    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345153    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345155    2580 flags.go:64] FLAG: --v="2"
Sep 10 21:19:20.353384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345159    2580 flags.go:64] FLAG: --version="false"
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345163    2580 flags.go:64] FLAG: --vmodule=""
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345167    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345170    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345309    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345313    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345316    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345319    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345323    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345326    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345329    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345332    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345335    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345337    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345341    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345346    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345349    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345352    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345355    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345358    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 21:19:20.354467 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345361    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345364    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345366    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345369    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345372    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345376    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345380    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345383    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345386    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345389    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345392    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345394    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345397    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345399    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345402    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345404    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345407    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345410    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345413    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345415    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 21:19:20.355334 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345418    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345420    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345423    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345425    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345428    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345430    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345433    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345437    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345439    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345442    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345445    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345447    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345450    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345452    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345455    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345457    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345460    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345462    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345464    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345475    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 21:19:20.355843 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345479    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345482    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345485    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345488    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345490    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345493    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345496    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345498    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345501    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345504    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345507    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345510    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345512    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345515    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345518    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345520    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345523    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345526    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345528    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345532    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 21:19:20.356798 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345535    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345538    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345541    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345544    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345546    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345549    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345552    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345555    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345557    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345560    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345563    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345567    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345572    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345575    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345578    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345582    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345585    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345587    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345590    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 21:19:20.357621 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345593    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345595    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345598    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.345601    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.345606    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.355244    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.355263    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.356000    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.356017    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356106    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356113    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356118    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356123    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356128    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356132    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 21:19:20.358261 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356137    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356141    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356145    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356149    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356154    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356158    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356163    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356167    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356171    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356175    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356181    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356185    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356189    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356193    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356197    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356202    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356206    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356210    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356214    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356218    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 21:19:20.358937 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356222    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356226    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356233    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356240    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356245    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356249    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356253    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356258    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356265    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356270    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356275    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356280    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356303    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356307    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356311    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356315    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356320    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356324    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356327    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 21:19:20.359623 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356332    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356335    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356339    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356343    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356349    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356352    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356357    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356361    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356365    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356369    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356373    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356377    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356381    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356386    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356390    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356395    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356399    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356404    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356408    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356412    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 21:19:20.360205 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356416    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356420    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356424    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356428    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356431    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356435    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356440    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356443    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356448    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356452    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356456    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356461    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356465    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356469    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356474    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356478    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356482    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356487    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356491    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356496    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 21:19:20.360719 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356500    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356504    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356509    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356513    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356517    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356521    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356528    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356533    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356539    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356544    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356549    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356553    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356558    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356562    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 21:19:20.361272 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.356570    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356749    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356757    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356763    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356767    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356773    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356777    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356782    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356786    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356790    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356794    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356798    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356802    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356806    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356810    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356814    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356818    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356823    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356827    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356831    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 21:19:20.361716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356835    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356839    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356843    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356847    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356851    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356855    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356860    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356864    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356869    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356873    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356878    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356883    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356889    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356896    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356901    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356907    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356911    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356915    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356920    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 21:19:20.362189 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356925    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356930    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356934    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356938    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356942    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356946    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356950    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356954    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356958    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356963    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356968    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356972    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356977    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356981    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356985    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356989    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356994    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.356998    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357002    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357006    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 21:19:20.362697 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357010    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357014    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357018    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357022    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357026    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357030    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357034    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357041    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357046    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357051    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357056    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357060    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357065    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357069    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357073    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357078    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357082    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357086    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357090    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 21:19:20.363173 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357094    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357098    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357103    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357107    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357112    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357117    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357121    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357125    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357129    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357134    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357138    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357142    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357147    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357151    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357155    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357159    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357163    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357167    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357171    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357175    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 21:19:20.363839 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357179    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 21:19:20.364759 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:20.357183    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 21:19:20.364759 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.357191    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 21:19:20.364759 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.357366    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 21:19:20.364759 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.359857    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 21:19:20.364759 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.360762    2580 server.go:1019] "Starting client certificate rotation"
Sep 10 21:19:20.364759 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.360864    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 21:19:20.364759 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.360894    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 21:19:20.365264 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.365234    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 21:19:20.367688 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.367666    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 21:19:20.377191 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.377174    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 10 21:19:20.380230 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.380214    2580 log.go:25] "Validated CRI v1 image API"
Sep 10 21:19:20.381112 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.381094    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 21:19:20.383200 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.383182    2580 fs.go:135] Filesystem UUIDs: map[6b1b51f0-5841-45b5-8e36-083c20660eab:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ac5a5da5-ee7b-4b1b-937a-fa5699298380:/dev/nvme0n1p4]
Sep 10 21:19:20.383258 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.383201    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 21:19:20.388334 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.388211    2580 manager.go:216] Machine: {Timestamp:2026-09-10 21:19:20.386917604 +0000 UTC m=+0.316956864 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091155 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec247a6c8d2223bef7574de85dbae652 SystemUUID:ec247a6c-8d22-23be-f757-4de85dbae652 BootID:19024035-9d67-43f4-b0cf-0d2cb70461b5 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:75:cc:3d:7c:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:75:cc:3d:7c:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:f2:e2:95:64:fd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 21:19:20.388334 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.388334    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 21:19:20.388427 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.388409    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 21:19:20.391205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.391190    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 21:19:20.391757 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.391737    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 21:19:20.391883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.391760    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 21:19:20.392822 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.392812    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 21:19:20.392864 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.392826    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 21:19:20.392864 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.392834    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 21:19:20.393468 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.393458    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 21:19:20.394138 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.394129    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 21:19:20.397976 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.397965    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 21:19:20.398092 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.398083    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 21:19:20.398121 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.398112    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 21:19:20.398149 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.398122    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 21:19:20.398149 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.398133    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 21:19:20.398149 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.398141    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 10 21:19:20.398149 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.398149    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 21:19:20.398517 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.398495    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 21:19:20.398579 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.398541    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 21:19:20.400814 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.400794    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 21:19:20.402552 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.402539    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 21:19:20.402607 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.402569    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 21:19:20.404244 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404231    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 21:19:20.404304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404249    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 21:19:20.404304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404255    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 21:19:20.404304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404261    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 21:19:20.404304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404269    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 21:19:20.404304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404279    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 21:19:20.404435 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404313    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 21:19:20.404435 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404319    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 21:19:20.404435 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404360    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 21:19:20.404435 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404369    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 21:19:20.404435 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404379    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 21:19:20.404435 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.404384    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 21:19:20.406395 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.406093    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 21:19:20.406484 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.406402    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 21:19:20.410097 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.410076    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-226.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 21:19:20.410528 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.410514    2580 server.go:1268] "Started kubelet"
Sep 10 21:19:20.410616 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.410594    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 21:19:20.410664 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.410580    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 21:19:20.410708 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.410632    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 21:19:20.410708 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.410692    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 21:19:20.410802 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.410595    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 21:19:20.411518 ip-10-0-135-226 systemd[1]: Started Kubernetes Kubelet.
Sep 10 21:19:20.412793 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.412775    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 21:19:20.413036 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.413024    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 21:19:20.415479 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.415462    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 21:19:20.415761 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.415743    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8h5b9"
Sep 10 21:19:20.416855 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.416843    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 21:19:20.417506 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.417491    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 21:19:20.417691 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.417678    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:20.418059 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418040    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 21:19:20.418059 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418045    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 21:19:20.418201 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418068    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 21:19:20.418201 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418189    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 21:19:20.418201 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418195    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 21:19:20.418563 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418544    2580 factory.go:153] Registering CRI-O factory
Sep 10 21:19:20.418615 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418571    2580 factory.go:223] Registration of the crio container factory successfully
Sep 10 21:19:20.418668 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418616    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 21:19:20.418668 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418623    2580 factory.go:55] Registering systemd factory
Sep 10 21:19:20.418668 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418629    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 10 21:19:20.418668 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418643    2580 factory.go:103] Registering Raw factory
Sep 10 21:19:20.418668 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.418655    2580 manager.go:1196] Started watching for new ooms in manager
Sep 10 21:19:20.419476 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.418615    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-226.ec2.internal.18d4126124bdfe20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-226.ec2.internal,UID:ip-10-0-135-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-226.ec2.internal,},FirstTimestamp:2026-09-10 21:19:20.410488352 +0000 UTC m=+0.340527613,LastTimestamp:2026-09-10 21:19:20.410488352 +0000 UTC m=+0.340527613,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-226.ec2.internal,}"
Sep 10 21:19:20.420070 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.420057    2580 manager.go:349] Starting recovery of all containers
Sep 10 21:19:20.425667 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.425511    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8h5b9"
Sep 10 21:19:20.425667 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.425615    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 21:19:20.425798 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.425638    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 21:19:20.430033 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.430014    2580 manager.go:354] Recovery completed
Sep 10 21:19:20.436171 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.436158    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 21:19:20.440266 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.440249    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 21:19:20.440333 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.440281    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 21:19:20.440333 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.440311    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Sep 10 21:19:20.440819 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.440803    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 21:19:20.440819 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.440815    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 21:19:20.440931 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.440816    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 21:19:20.440931 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.440882    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 21:19:20.441616 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.441600    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 21:19:20.441678 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.441635    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 21:19:20.441678 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.441649    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Sep 10 21:19:20.442125 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.442063    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-226.ec2.internal.18d41261268461c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-226.ec2.internal,UID:ip-10-0-135-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-226.ec2.internal,},FirstTimestamp:2026-09-10 21:19:20.440267208 +0000 UTC m=+0.370306473,LastTimestamp:2026-09-10 21:19:20.440267208 +0000 UTC m=+0.370306473,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-226.ec2.internal,}"
Sep 10 21:19:20.442239 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.442226    2580 policy_none.go:49] "None policy: Start"
Sep 10 21:19:20.442309 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.442243    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 21:19:20.442309 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.442253    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 21:19:20.443407 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.443396    2580 policy_none.go:47] "Start"
Sep 10 21:19:20.485890 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.485874    2580 manager.go:337] "Starting Device Plugin manager"
Sep 10 21:19:20.500064 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.485910    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 21:19:20.500064 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.485920    2580 server.go:87] "Starting device plugin registration server"
Sep 10 21:19:20.500064 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.486220    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 21:19:20.500064 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.486230    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 21:19:20.500064 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.486366    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 21:19:20.500064 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.486424    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 21:19:20.500064 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.486429    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 21:19:20.500064 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.487053    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 21:19:20.500064 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.487104    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:20.530180 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.530144    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 21:19:20.531548 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.531534    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 21:19:20.531603 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.531580    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 21:19:20.531638 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.531603    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 21:19:20.531676 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.531641    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 21:19:20.534315 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.534278    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 21:19:20.586720 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.586686    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 21:19:20.587909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.587837    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 21:19:20.587909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.587879    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 21:19:20.587909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.587889    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Sep 10 21:19:20.587909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.587913    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.596150 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.596132    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.596230 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.596158    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-226.ec2.internal\": node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:20.610975 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.610950    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:20.632642 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.632600    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"]
Sep 10 21:19:20.632728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.632692    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 21:19:20.634221 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.634206    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 21:19:20.634283 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.634236    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 21:19:20.634283 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.634246    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Sep 10 21:19:20.635552 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.635538    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 21:19:20.635659 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.635645    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.635707 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.635678    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 21:19:20.636299 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.636273    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 21:19:20.636353 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.636322    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 21:19:20.636353 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.636333    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Sep 10 21:19:20.636353 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.636276    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 21:19:20.636441 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.636368    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 21:19:20.636441 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.636377    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Sep 10 21:19:20.637577 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.637560    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.637669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.637589    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 21:19:20.638276 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.638259    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 21:19:20.638359 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.638312    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 21:19:20.638359 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.638329    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Sep 10 21:19:20.661838 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.661822    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-226.ec2.internal\" not found" node="ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.665651 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.665630    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-226.ec2.internal\" not found" node="ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.711305 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.711268    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:20.719196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.719182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30a08eec8551a0009e61c657650503a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"30a08eec8551a0009e61c657650503a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.719257 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.719203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30a08eec8551a0009e61c657650503a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"30a08eec8551a0009e61c657650503a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.719257 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.719219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d6ceb36f4f0ad1940c1c73fe62eb8fd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-226.ec2.internal\" (UID: \"2d6ceb36f4f0ad1940c1c73fe62eb8fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.811723 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.811700    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:20.819973 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.819956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30a08eec8551a0009e61c657650503a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"30a08eec8551a0009e61c657650503a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.820032 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.820001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30a08eec8551a0009e61c657650503a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"30a08eec8551a0009e61c657650503a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.820066 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.820044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30a08eec8551a0009e61c657650503a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"30a08eec8551a0009e61c657650503a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.820114 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.820102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d6ceb36f4f0ad1940c1c73fe62eb8fd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-226.ec2.internal\" (UID: \"2d6ceb36f4f0ad1940c1c73fe62eb8fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.820148 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.820045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30a08eec8551a0009e61c657650503a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"30a08eec8551a0009e61c657650503a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.820148 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.820130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d6ceb36f4f0ad1940c1c73fe62eb8fd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-226.ec2.internal\" (UID: \"2d6ceb36f4f0ad1940c1c73fe62eb8fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.912459 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:20.912384    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:20.963842 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.963811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:20.967845 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:20.967778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:21.013300 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.013265    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:21.113755 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.113729    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:21.214313 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.214233    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:21.314851 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.314819    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:21.360968 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:21.360945    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 21:19:21.361459 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:21.361059    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 21:19:21.361459 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.361087    2580 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.135.226:47862->34.192.161.81:6443: use of closed network connection"
Sep 10 21:19:21.361459 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.361168    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: unexpected error when reading response body. Please retry. Original error: read tcp 10.0.135.226:47862->34.192.161.81:6443: use of closed network connection" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 21:19:21.380241 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:21.380221    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 21:19:21.415255 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.415217    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:21.435067 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:21.435021    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 21:14:21 +0000 UTC" deadline="2028-02-05 18:31:03.183291401 +0000 UTC"
Sep 10 21:19:21.435067 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:21.435063    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12309h11m41.748232383s"
Sep 10 21:19:21.471208 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:21.471136    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6ceb36f4f0ad1940c1c73fe62eb8fd.slice/crio-3c9d1e6f0c95db21d101900193add1c16991128431859b2e8fda384f8247911a WatchSource:0}: Error finding container 3c9d1e6f0c95db21d101900193add1c16991128431859b2e8fda384f8247911a: Status 404 returned error can't find the container with id 3c9d1e6f0c95db21d101900193add1c16991128431859b2e8fda384f8247911a
Sep 10 21:19:21.475726 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:21.475694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a08eec8551a0009e61c657650503a2.slice/crio-03e81767ebb6804c90c8560dd7aa96964bee470308a5e5c8593951ca55f9a893 WatchSource:0}: Error finding container 03e81767ebb6804c90c8560dd7aa96964bee470308a5e5c8593951ca55f9a893: Status 404 returned error can't find the container with id 03e81767ebb6804c90c8560dd7aa96964bee470308a5e5c8593951ca55f9a893
Sep 10 21:19:21.476257 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:21.476237    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:19:21.515689 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.515656    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:21.535459 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:21.535402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal" event={"ID":"30a08eec8551a0009e61c657650503a2","Type":"ContainerStarted","Data":"03e81767ebb6804c90c8560dd7aa96964bee470308a5e5c8593951ca55f9a893"}
Sep 10 21:19:21.536388 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:21.536369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal" event={"ID":"2d6ceb36f4f0ad1940c1c73fe62eb8fd","Type":"ContainerStarted","Data":"3c9d1e6f0c95db21d101900193add1c16991128431859b2e8fda384f8247911a"}
Sep 10 21:19:21.616594 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.616563    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:21.717071 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.717040    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:21.817521 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.817451    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:21.918195 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:21.918162    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Sep 10 21:19:23.256343 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.256315    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 21:19:23.300293 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.300264    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:23.312677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.312658    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 21:19:23.314091 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.314070    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Sep 10 21:19:23.322970 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.322951    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 21:19:23.681743 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.681713    2580 apiserver.go:52] "Watching apiserver"
Sep 10 21:19:23.690695 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.690650    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 21:19:23.692448 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.692401    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-9txsp","openshift-dns/node-resolver-zzrvw","openshift-network-diagnostics/network-check-target-dz7bd","openshift-network-operator/iptables-alerter-hxssh","kube-system/konnectivity-agent-zs4b5","openshift-image-registry/node-ca-lj6vm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal","openshift-multus/multus-additional-cni-plugins-cxxt6","openshift-multus/multus-m2mjz","openshift-multus/network-metrics-daemon-kgjnr","openshift-ovn-kubernetes/ovnkube-node-cq76m","kube-system/global-pull-secret-syncer-2r6cp","kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"]
Sep 10 21:19:23.694278 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.694246    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:23.694381 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:23.694354    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:23.695706 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.695686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.696962 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.696941    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:23.697057 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:23.697022    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:23.698891 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.697803    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 21:19:23.698891 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.697946    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 21:19:23.698891 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.698267    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 21:19:23.698891 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.698482    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bgsjv\""
Sep 10 21:19:23.698891 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.698578    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 21:19:23.698891 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.698771    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 21:19:23.699216 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.698939    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 21:19:23.699216 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.699194    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hxssh"
Sep 10 21:19:23.701364 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.700040    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 21:19:23.701364 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.700695    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:23.701364 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.701225    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:19:23.701364 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.701233    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 21:19:23.701619 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.701496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lj6vm"
Sep 10 21:19:23.702032 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.702013    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 21:19:23.702401 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.702385    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jx2q9\""
Sep 10 21:19:23.702747 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.702729    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.702830 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.702804    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 21:19:23.703004 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.702989    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 21:19:23.703246 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.703230    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gtj6t\""
Sep 10 21:19:23.703605 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.703586    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 21:19:23.703769 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.703749    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 21:19:23.703842 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.703779    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 21:19:23.703895 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.703786    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8p5kl\""
Sep 10 21:19:23.704009 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.703994    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.705380 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.705086    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.705380 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.705350    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 21:19:23.705565 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.705549    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kqmkk\""
Sep 10 21:19:23.705658 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.705638    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 21:19:23.705716 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.705688    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 21:19:23.705789 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.705772    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 21:19:23.706331 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.706313    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 21:19:23.706331 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.706325    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 21:19:23.706513 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.706317    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 21:19:23.706513 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.706318    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cjcs4\""
Sep 10 21:19:23.706736 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.706715    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.707270 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.707251    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 21:19:23.707392 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.707369    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4wggk\""
Sep 10 21:19:23.707553 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.707534    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 21:19:23.708149 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.708130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:23.708241 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:23.708192    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:23.708916 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.708873    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:19:23.709037 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.709020    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 21:19:23.709161 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.709143    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7fcbw\""
Sep 10 21:19:23.710008 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.709852    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 21:19:23.710289 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.710271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zzrvw"
Sep 10 21:19:23.711554 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-cni-bin\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.711644 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f067fda-a406-4cf0-9135-b1efa82f373f-serviceca\") pod \"node-ca-lj6vm\" (UID: \"3f067fda-a406-4cf0-9135-b1efa82f373f\") " pod="openshift-image-registry/node-ca-lj6vm"
Sep 10 21:19:23.711644 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-systemd-units\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.711760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-var-lib-openvswitch\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.711760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-node-log\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.711760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.711760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e22316c-8ace-43e8-a93f-2e7272222b9c-ovnkube-config\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.711959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4b5\" (UniqueName: \"kubernetes.io/projected/6e22316c-8ace-43e8-a93f-2e7272222b9c-kube-api-access-cp4b5\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.711959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a3eda3e-fd65-4cab-b5f2-2870340b88b4-host-slash\") pod \"iptables-alerter-hxssh\" (UID: \"3a3eda3e-fd65-4cab-b5f2-2870340b88b4\") " pod="openshift-network-operator/iptables-alerter-hxssh"
Sep 10 21:19:23.711959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d2f5710-7e64-49ab-a1c1-6d5cce892e5e-konnectivity-ca\") pod \"konnectivity-agent-zs4b5\" (UID: \"6d2f5710-7e64-49ab-a1c1-6d5cce892e5e\") " pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:23.711959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqf4r\" (UniqueName: \"kubernetes.io/projected/3a3eda3e-fd65-4cab-b5f2-2870340b88b4-kube-api-access-nqf4r\") pod \"iptables-alerter-hxssh\" (UID: \"3a3eda3e-fd65-4cab-b5f2-2870340b88b4\") " pod="openshift-network-operator/iptables-alerter-hxssh"
Sep 10 21:19:23.711959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cqf5\" (UniqueName: \"kubernetes.io/projected/51decde2-f81d-4c6f-b857-8fd6150f288a-kube-api-access-4cqf5\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:23.711959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-kubelet\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.711959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-run-netns\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.711973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-etc-openvswitch\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e22316c-8ace-43e8-a93f-2e7272222b9c-env-overrides\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-slash\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-run-systemd\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-cni-netd\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e22316c-8ace-43e8-a93f-2e7272222b9c-ovn-node-metrics-cert\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbh2\" (UniqueName: \"kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2\") pod \"network-check-target-dz7bd\" (UID: \"9dca27a7-8d1a-4c60-bacf-bfad74056541\") " pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:23.712274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f067fda-a406-4cf0-9135-b1efa82f373f-host\") pod \"node-ca-lj6vm\" (UID: \"3f067fda-a406-4cf0-9135-b1efa82f373f\") " pod="openshift-image-registry/node-ca-lj6vm"
Sep 10 21:19:23.712274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:23.712274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a3eda3e-fd65-4cab-b5f2-2870340b88b4-iptables-alerter-script\") pod \"iptables-alerter-hxssh\" (UID: \"3a3eda3e-fd65-4cab-b5f2-2870340b88b4\") " pod="openshift-network-operator/iptables-alerter-hxssh"
Sep 10 21:19:23.712892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-log-socket\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e22316c-8ace-43e8-a93f-2e7272222b9c-ovnkube-script-lib\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d2f5710-7e64-49ab-a1c1-6d5cce892e5e-agent-certs\") pod \"konnectivity-agent-zs4b5\" (UID: \"6d2f5710-7e64-49ab-a1c1-6d5cce892e5e\") " pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:23.712892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj6k8\" (UniqueName: \"kubernetes.io/projected/3f067fda-a406-4cf0-9135-b1efa82f373f-kube-api-access-nj6k8\") pod \"node-ca-lj6vm\" (UID: \"3f067fda-a406-4cf0-9135-b1efa82f373f\") " pod="openshift-image-registry/node-ca-lj6vm"
Sep 10 21:19:23.712892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-run-ovn\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-run-openvswitch\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.712892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712604    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4kn88\""
Sep 10 21:19:23.712892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712572    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 21:19:23.712892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.712726    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 21:19:23.716681 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.716662    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 21:19:23.736645 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.736624    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qqmkc"
Sep 10 21:19:23.743911 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.743890    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qqmkc"
Sep 10 21:19:23.801278 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.801248    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 21:19:23.813582 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqf4r\" (UniqueName: \"kubernetes.io/projected/3a3eda3e-fd65-4cab-b5f2-2870340b88b4-kube-api-access-nqf4r\") pod \"iptables-alerter-hxssh\" (UID: \"3a3eda3e-fd65-4cab-b5f2-2870340b88b4\") " pod="openshift-network-operator/iptables-alerter-hxssh"
Sep 10 21:19:23.813685 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed0bb520-aceb-44a9-8796-178e661d1482-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.813685 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-lib-modules\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.813685 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-var-lib-kubelet\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.813685 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-host\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.813883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-tuned\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.813883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb4xs\" (UniqueName: \"kubernetes.io/projected/1e48ee8f-64c3-4ab2-950c-f09d5043d472-kube-api-access-zb4xs\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.813883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-run-netns\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.813883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e22316c-8ace-43e8-a93f-2e7272222b9c-env-overrides\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.813883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7bs\" (UniqueName: \"kubernetes.io/projected/ed0bb520-aceb-44a9-8796-178e661d1482-kube-api-access-qp7bs\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.813883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5q6p\" (UniqueName: \"kubernetes.io/projected/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-kube-api-access-r5q6p\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.813883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-cni-netd\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.813883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e22316c-8ace-43e8-a93f-2e7272222b9c-ovn-node-metrics-cert\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.813883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbh2\" (UniqueName: \"kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2\") pod \"network-check-target-dz7bd\" (UID: \"9dca27a7-8d1a-4c60-bacf-bfad74056541\") " pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:23.813883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f067fda-a406-4cf0-9135-b1efa82f373f-host\") pod \"node-ca-lj6vm\" (UID: \"3f067fda-a406-4cf0-9135-b1efa82f373f\") " pod="openshift-image-registry/node-ca-lj6vm"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-modprobe-d\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-sysctl-conf\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-run\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkfhr\" (UniqueName: \"kubernetes.io/projected/eef3bddd-76e7-4f9f-867c-1af56e489709-kube-api-access-gkfhr\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.813974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e804440-819b-402d-ba84-f9f0c3ad4aeb-kubelet-config\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e804440-819b-402d-ba84-f9f0c3ad4aeb-dbus\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6987485a-cc5d-45a1-a8e4-1eb4cf948766-tmp-dir\") pod \"node-resolver-zzrvw\" (UID: \"6987485a-cc5d-45a1-a8e4-1eb4cf948766\") " pod="openshift-dns/node-resolver-zzrvw"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-systemd\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-sys\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-hostroot\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.814236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-registration-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed0bb520-aceb-44a9-8796-178e661d1482-system-cni-dir\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-sysctl-d\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-systemd-units\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-var-lib-openvswitch\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e22316c-8ace-43e8-a93f-2e7272222b9c-ovnkube-config\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4b5\" (UniqueName: \"kubernetes.io/projected/6e22316c-8ace-43e8-a93f-2e7272222b9c-kube-api-access-cp4b5\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-system-cni-dir\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-var-lib-kubelet\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-conf-dir\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-kubernetes\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-cni-dir\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-var-lib-cni-multus\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-etc-kubernetes\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6987485a-cc5d-45a1-a8e4-1eb4cf948766-hosts-file\") pod \"node-resolver-zzrvw\" (UID: \"6987485a-cc5d-45a1-a8e4-1eb4cf948766\") " pod="openshift-dns/node-resolver-zzrvw"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cqf5\" (UniqueName: \"kubernetes.io/projected/51decde2-f81d-4c6f-b857-8fd6150f288a-kube-api-access-4cqf5\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:23.814762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-kubelet\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-etc-openvswitch\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed0bb520-aceb-44a9-8796-178e661d1482-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-run-multus-certs\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-sys-fs\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-slash\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-run-systemd\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-socket-dir-parent\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a3eda3e-fd65-4cab-b5f2-2870340b88b4-iptables-alerter-script\") pod \"iptables-alerter-hxssh\" (UID: \"3a3eda3e-fd65-4cab-b5f2-2870340b88b4\") " pod="openshift-network-operator/iptables-alerter-hxssh"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed0bb520-aceb-44a9-8796-178e661d1482-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed0bb520-aceb-44a9-8796-178e661d1482-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814964    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-run-k8s-cni-cncf-io\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.814988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-var-lib-cni-bin\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sggt2\" (UniqueName: \"kubernetes.io/projected/6987485a-cc5d-45a1-a8e4-1eb4cf948766-kube-api-access-sggt2\") pod \"node-resolver-zzrvw\" (UID: \"6987485a-cc5d-45a1-a8e4-1eb4cf948766\") " pod="openshift-dns/node-resolver-zzrvw"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-log-socket\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.815335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e22316c-8ace-43e8-a93f-2e7272222b9c-ovnkube-script-lib\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d2f5710-7e64-49ab-a1c1-6d5cce892e5e-agent-certs\") pod \"konnectivity-agent-zs4b5\" (UID: \"6d2f5710-7e64-49ab-a1c1-6d5cce892e5e\") " pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj6k8\" (UniqueName: \"kubernetes.io/projected/3f067fda-a406-4cf0-9135-b1efa82f373f-kube-api-access-nj6k8\") pod \"node-ca-lj6vm\" (UID: \"3f067fda-a406-4cf0-9135-b1efa82f373f\") " pod="openshift-image-registry/node-ca-lj6vm"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eef3bddd-76e7-4f9f-867c-1af56e489709-tmp\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-socket-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-run-ovn\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-run-openvswitch\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-cni-bin\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-run-netns\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-device-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f067fda-a406-4cf0-9135-b1efa82f373f-serviceca\") pod \"node-ca-lj6vm\" (UID: \"3f067fda-a406-4cf0-9135-b1efa82f373f\") " pod="openshift-image-registry/node-ca-lj6vm"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-daemon-config\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-node-log\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a3eda3e-fd65-4cab-b5f2-2870340b88b4-host-slash\") pod \"iptables-alerter-hxssh\" (UID: \"3a3eda3e-fd65-4cab-b5f2-2870340b88b4\") " pod="openshift-network-operator/iptables-alerter-hxssh"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d2f5710-7e64-49ab-a1c1-6d5cce892e5e-konnectivity-ca\") pod \"konnectivity-agent-zs4b5\" (UID: \"6d2f5710-7e64-49ab-a1c1-6d5cce892e5e\") " pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed0bb520-aceb-44a9-8796-178e661d1482-cnibin\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.815920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-sysconfig\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.816536 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-cnibin\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.816536 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.815649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e48ee8f-64c3-4ab2-950c-f09d5043d472-cni-binary-copy\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.816536 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.816031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-run-netns\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.816536 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.816515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e22316c-8ace-43e8-a93f-2e7272222b9c-env-overrides\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.816679 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.816609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-cni-netd\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.816888 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.816861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-etc-openvswitch\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.817197 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.817161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f067fda-a406-4cf0-9135-b1efa82f373f-host\") pod \"node-ca-lj6vm\" (UID: \"3f067fda-a406-4cf0-9135-b1efa82f373f\") " pod="openshift-image-registry/node-ca-lj6vm"
Sep 10 21:19:23.817286 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:23.817270    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:23.817359 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.817280    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 21:19:23.817359 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:23.817340    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs podName:51decde2-f81d-4c6f-b857-8fd6150f288a nodeName:}" failed. No retries permitted until 2026-09-10 21:19:24.317308552 +0000 UTC m=+2.965222715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs") pod "network-metrics-daemon-kgjnr" (UID: "51decde2-f81d-4c6f-b857-8fd6150f288a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:23.817593 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.817564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.817953 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.817928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-kubelet\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.818071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.817943    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-run-ovn\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.818071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.817992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-slash\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.818071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-log-socket\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.818071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-run-systemd\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.818296 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.818296 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-var-lib-openvswitch\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.818296 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-node-log\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.818296 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-systemd-units\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.818527 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f067fda-a406-4cf0-9135-b1efa82f373f-serviceca\") pod \"node-ca-lj6vm\" (UID: \"3f067fda-a406-4cf0-9135-b1efa82f373f\") " pod="openshift-image-registry/node-ca-lj6vm"
Sep 10 21:19:23.818527 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a3eda3e-fd65-4cab-b5f2-2870340b88b4-iptables-alerter-script\") pod \"iptables-alerter-hxssh\" (UID: \"3a3eda3e-fd65-4cab-b5f2-2870340b88b4\") " pod="openshift-network-operator/iptables-alerter-hxssh"
Sep 10 21:19:23.818667 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e22316c-8ace-43e8-a93f-2e7272222b9c-ovnkube-script-lib\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.818667 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-run-openvswitch\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.818667 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818636    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a3eda3e-fd65-4cab-b5f2-2870340b88b4-host-slash\") pod \"iptables-alerter-hxssh\" (UID: \"3a3eda3e-fd65-4cab-b5f2-2870340b88b4\") " pod="openshift-network-operator/iptables-alerter-hxssh"
Sep 10 21:19:23.818667 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e22316c-8ace-43e8-a93f-2e7272222b9c-host-cni-bin\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.819008 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.818985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d2f5710-7e64-49ab-a1c1-6d5cce892e5e-konnectivity-ca\") pod \"konnectivity-agent-zs4b5\" (UID: \"6d2f5710-7e64-49ab-a1c1-6d5cce892e5e\") " pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:23.819746 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.819562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e22316c-8ace-43e8-a93f-2e7272222b9c-ovnkube-config\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.823232 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.823213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e22316c-8ace-43e8-a93f-2e7272222b9c-ovn-node-metrics-cert\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.824735 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:23.824719    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 21:19:23.824853 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:23.824737    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 21:19:23.824965 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:23.824863    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rxbh2 for pod openshift-network-diagnostics/network-check-target-dz7bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:23.824965 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:23.824912    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2 podName:9dca27a7-8d1a-4c60-bacf-bfad74056541 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:24.324898653 +0000 UTC m=+2.972812812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rxbh2" (UniqueName: "kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2") pod "network-check-target-dz7bd" (UID: "9dca27a7-8d1a-4c60-bacf-bfad74056541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:23.824965 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.824748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqf4r\" (UniqueName: \"kubernetes.io/projected/3a3eda3e-fd65-4cab-b5f2-2870340b88b4-kube-api-access-nqf4r\") pod \"iptables-alerter-hxssh\" (UID: \"3a3eda3e-fd65-4cab-b5f2-2870340b88b4\") " pod="openshift-network-operator/iptables-alerter-hxssh"
Sep 10 21:19:23.828045 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.828019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4b5\" (UniqueName: \"kubernetes.io/projected/6e22316c-8ace-43e8-a93f-2e7272222b9c-kube-api-access-cp4b5\") pod \"ovnkube-node-cq76m\" (UID: \"6e22316c-8ace-43e8-a93f-2e7272222b9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:23.828169 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.828148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj6k8\" (UniqueName: \"kubernetes.io/projected/3f067fda-a406-4cf0-9135-b1efa82f373f-kube-api-access-nj6k8\") pod \"node-ca-lj6vm\" (UID: \"3f067fda-a406-4cf0-9135-b1efa82f373f\") " pod="openshift-image-registry/node-ca-lj6vm"
Sep 10 21:19:23.828612 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.828589    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cqf5\" (UniqueName: \"kubernetes.io/projected/51decde2-f81d-4c6f-b857-8fd6150f288a-kube-api-access-4cqf5\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:23.831954 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.831932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d2f5710-7e64-49ab-a1c1-6d5cce892e5e-agent-certs\") pod \"konnectivity-agent-zs4b5\" (UID: \"6d2f5710-7e64-49ab-a1c1-6d5cce892e5e\") " pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:23.916857 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.916829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sggt2\" (UniqueName: \"kubernetes.io/projected/6987485a-cc5d-45a1-a8e4-1eb4cf948766-kube-api-access-sggt2\") pod \"node-resolver-zzrvw\" (UID: \"6987485a-cc5d-45a1-a8e4-1eb4cf948766\") " pod="openshift-dns/node-resolver-zzrvw"
Sep 10 21:19:23.916857 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.916867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eef3bddd-76e7-4f9f-867c-1af56e489709-tmp\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.917087 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.916891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-socket-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.917087 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.916912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:23.917087 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.916935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-run-netns\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.917087 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.916954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-device-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.917087 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.916976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-daemon-config\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.917087 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed0bb520-aceb-44a9-8796-178e661d1482-cnibin\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.917087 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-sysconfig\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.917087 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-cnibin\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.917087 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-socket-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.917087 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e48ee8f-64c3-4ab2-950c-f09d5043d472-cni-binary-copy\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed0bb520-aceb-44a9-8796-178e661d1482-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-lib-modules\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-var-lib-kubelet\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-run-netns\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-host\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-tuned\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb4xs\" (UniqueName: \"kubernetes.io/projected/1e48ee8f-64c3-4ab2-950c-f09d5043d472-kube-api-access-zb4xs\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7bs\" (UniqueName: \"kubernetes.io/projected/ed0bb520-aceb-44a9-8796-178e661d1482-kube-api-access-qp7bs\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5q6p\" (UniqueName: \"kubernetes.io/projected/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-kube-api-access-r5q6p\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-sysconfig\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-modprobe-d\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-sysctl-conf\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-cnibin\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-run\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed0bb520-aceb-44a9-8796-178e661d1482-cnibin\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkfhr\" (UniqueName: \"kubernetes.io/projected/eef3bddd-76e7-4f9f-867c-1af56e489709-kube-api-access-gkfhr\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.917576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-device-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e804440-819b-402d-ba84-f9f0c3ad4aeb-kubelet-config\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e804440-819b-402d-ba84-f9f0c3ad4aeb-dbus\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6987485a-cc5d-45a1-a8e4-1eb4cf948766-tmp-dir\") pod \"node-resolver-zzrvw\" (UID: \"6987485a-cc5d-45a1-a8e4-1eb4cf948766\") " pod="openshift-dns/node-resolver-zzrvw"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-systemd\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-sys\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-hostroot\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-daemon-config\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-registration-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed0bb520-aceb-44a9-8796-178e661d1482-system-cni-dir\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-systemd\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-sysctl-d\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-system-cni-dir\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-var-lib-kubelet\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-conf-dir\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e804440-819b-402d-ba84-f9f0c3ad4aeb-kubelet-config\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:23.918279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-kubernetes\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e804440-819b-402d-ba84-f9f0c3ad4aeb-dbus\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-cni-dir\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-sys\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-var-lib-cni-multus\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-var-lib-cni-multus\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-etc-kubernetes\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6987485a-cc5d-45a1-a8e4-1eb4cf948766-hosts-file\") pod \"node-resolver-zzrvw\" (UID: \"6987485a-cc5d-45a1-a8e4-1eb4cf948766\") " pod="openshift-dns/node-resolver-zzrvw"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-modprobe-d\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-registration-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-hostroot\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed0bb520-aceb-44a9-8796-178e661d1482-system-cni-dir\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-lib-modules\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-var-lib-kubelet\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.917972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed0bb520-aceb-44a9-8796-178e661d1482-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-sysctl-d\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-host\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-run-multus-certs\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-sys-fs\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-system-cni-dir\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-run-multus-certs\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-sys-fs\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed0bb520-aceb-44a9-8796-178e661d1482-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:23.918663    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-var-lib-kubelet\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-sysctl-conf\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-conf-dir\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed0bb520-aceb-44a9-8796-178e661d1482-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-kubernetes\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eef3bddd-76e7-4f9f-867c-1af56e489709-run\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-socket-dir-parent\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6987485a-cc5d-45a1-a8e4-1eb4cf948766-tmp-dir\") pod \"node-resolver-zzrvw\" (UID: \"6987485a-cc5d-45a1-a8e4-1eb4cf948766\") " pod="openshift-dns/node-resolver-zzrvw"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:23.918828    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret podName:7e804440-819b-402d-ba84-f9f0c3ad4aeb nodeName:}" failed. No retries permitted until 2026-09-10 21:19:24.418803085 +0000 UTC m=+3.066717259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret") pod "global-pull-secret-syncer-2r6cp" (UID: "7e804440-819b-402d-ba84-f9f0c3ad4aeb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-etc-kubernetes\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed0bb520-aceb-44a9-8796-178e661d1482-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.919963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-socket-dir-parent\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6987485a-cc5d-45a1-a8e4-1eb4cf948766-hosts-file\") pod \"node-resolver-zzrvw\" (UID: \"6987485a-cc5d-45a1-a8e4-1eb4cf948766\") " pod="openshift-dns/node-resolver-zzrvw"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.918880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-multus-cni-dir\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.919065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed0bb520-aceb-44a9-8796-178e661d1482-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.919102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-run-k8s-cni-cncf-io\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.919157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-var-lib-cni-bin\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.919162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-run-k8s-cni-cncf-io\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.919201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.919264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e48ee8f-64c3-4ab2-950c-f09d5043d472-host-var-lib-cni-bin\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.919312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.919449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed0bb520-aceb-44a9-8796-178e661d1482-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.919613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e48ee8f-64c3-4ab2-950c-f09d5043d472-cni-binary-copy\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.919643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eef3bddd-76e7-4f9f-867c-1af56e489709-tmp\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.920760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.920553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eef3bddd-76e7-4f9f-867c-1af56e489709-etc-tuned\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.921532 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.921510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed0bb520-aceb-44a9-8796-178e661d1482-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.925468 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.925446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sggt2\" (UniqueName: \"kubernetes.io/projected/6987485a-cc5d-45a1-a8e4-1eb4cf948766-kube-api-access-sggt2\") pod \"node-resolver-zzrvw\" (UID: \"6987485a-cc5d-45a1-a8e4-1eb4cf948766\") " pod="openshift-dns/node-resolver-zzrvw"
Sep 10 21:19:23.925964 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.925935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7bs\" (UniqueName: \"kubernetes.io/projected/ed0bb520-aceb-44a9-8796-178e661d1482-kube-api-access-qp7bs\") pod \"multus-additional-cni-plugins-cxxt6\" (UID: \"ed0bb520-aceb-44a9-8796-178e661d1482\") " pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:23.926087 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.926036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkfhr\" (UniqueName: \"kubernetes.io/projected/eef3bddd-76e7-4f9f-867c-1af56e489709-kube-api-access-gkfhr\") pod \"tuned-9txsp\" (UID: \"eef3bddd-76e7-4f9f-867c-1af56e489709\") " pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:23.926368 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.926351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5q6p\" (UniqueName: \"kubernetes.io/projected/e8581684-62f8-4d33-bce5-4e3a1e7a25e4-kube-api-access-r5q6p\") pod \"aws-ebs-csi-driver-node-829ln\" (UID: \"e8581684-62f8-4d33-bce5-4e3a1e7a25e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:23.927372 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:23.927350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb4xs\" (UniqueName: \"kubernetes.io/projected/1e48ee8f-64c3-4ab2-950c-f09d5043d472-kube-api-access-zb4xs\") pod \"multus-m2mjz\" (UID: \"1e48ee8f-64c3-4ab2-950c-f09d5043d472\") " pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:24.009612 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.009529    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:24.018284 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.018261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hxssh"
Sep 10 21:19:24.029735 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.029714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:24.035043 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.035023    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lj6vm"
Sep 10 21:19:24.042054 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.042028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m2mjz"
Sep 10 21:19:24.049013 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.048992    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln"
Sep 10 21:19:24.055763 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.055746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cxxt6"
Sep 10 21:19:24.062111 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.062092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9txsp"
Sep 10 21:19:24.067560 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.067543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zzrvw"
Sep 10 21:19:24.321668 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.321589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:24.321795 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:24.321718    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:24.321795 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:24.321779    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs podName:51decde2-f81d-4c6f-b857-8fd6150f288a nodeName:}" failed. No retries permitted until 2026-09-10 21:19:25.321762206 +0000 UTC m=+3.969676382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs") pod "network-metrics-daemon-kgjnr" (UID: "51decde2-f81d-4c6f-b857-8fd6150f288a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:24.402215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.402185    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 21:19:24.422709 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.422677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbh2\" (UniqueName: \"kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2\") pod \"network-check-target-dz7bd\" (UID: \"9dca27a7-8d1a-4c60-bacf-bfad74056541\") " pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:24.422857 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.422765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:24.422909 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:24.422853    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 21:19:24.422909 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:24.422872    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 21:19:24.422909 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:24.422881    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rxbh2 for pod openshift-network-diagnostics/network-check-target-dz7bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:24.423002 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:24.422925    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2 podName:9dca27a7-8d1a-4c60-bacf-bfad74056541 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:25.422912546 +0000 UTC m=+4.070826691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rxbh2" (UniqueName: "kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2") pod "network-check-target-dz7bd" (UID: "9dca27a7-8d1a-4c60-bacf-bfad74056541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:24.423073 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:24.422852    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:24.423181 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:24.423124    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret podName:7e804440-819b-402d-ba84-f9f0c3ad4aeb nodeName:}" failed. No retries permitted until 2026-09-10 21:19:25.423108539 +0000 UTC m=+4.071022701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret") pod "global-pull-secret-syncer-2r6cp" (UID: "7e804440-819b-402d-ba84-f9f0c3ad4aeb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:24.641909 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:24.641885    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f067fda_a406_4cf0_9135_b1efa82f373f.slice/crio-41c545e133e289ab8c31a8385dd1df3f72fc574444958633cda604cd455c014c WatchSource:0}: Error finding container 41c545e133e289ab8c31a8385dd1df3f72fc574444958633cda604cd455c014c: Status 404 returned error can't find the container with id 41c545e133e289ab8c31a8385dd1df3f72fc574444958633cda604cd455c014c
Sep 10 21:19:24.643462 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:24.643248    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2f5710_7e64_49ab_a1c1_6d5cce892e5e.slice/crio-c036ac63b4bbd575037cb7d70ecc922dda70bd8e35540f03556b35ffbb4ed175 WatchSource:0}: Error finding container c036ac63b4bbd575037cb7d70ecc922dda70bd8e35540f03556b35ffbb4ed175: Status 404 returned error can't find the container with id c036ac63b4bbd575037cb7d70ecc922dda70bd8e35540f03556b35ffbb4ed175
Sep 10 21:19:24.644359 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:24.644339    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6987485a_cc5d_45a1_a8e4_1eb4cf948766.slice/crio-6f8f62c3124c925eaad031b2d0fef92266d5f3fbb2f7206978269b0526415120 WatchSource:0}: Error finding container 6f8f62c3124c925eaad031b2d0fef92266d5f3fbb2f7206978269b0526415120: Status 404 returned error can't find the container with id 6f8f62c3124c925eaad031b2d0fef92266d5f3fbb2f7206978269b0526415120
Sep 10 21:19:24.645182 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:24.645115    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3eda3e_fd65_4cab_b5f2_2870340b88b4.slice/crio-753637f87299f0303555cf875c0946677fb2fca81baa1ce22030c5983f342245 WatchSource:0}: Error finding container 753637f87299f0303555cf875c0946677fb2fca81baa1ce22030c5983f342245: Status 404 returned error can't find the container with id 753637f87299f0303555cf875c0946677fb2fca81baa1ce22030c5983f342245
Sep 10 21:19:24.646901 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:24.646790    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0bb520_aceb_44a9_8796_178e661d1482.slice/crio-45e2f5f68a7558e822cb3f83df79bb729359d1a1c0c155d09a6c633a13c01862 WatchSource:0}: Error finding container 45e2f5f68a7558e822cb3f83df79bb729359d1a1c0c155d09a6c633a13c01862: Status 404 returned error can't find the container with id 45e2f5f68a7558e822cb3f83df79bb729359d1a1c0c155d09a6c633a13c01862
Sep 10 21:19:24.648497 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:24.648433    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8581684_62f8_4d33_bce5_4e3a1e7a25e4.slice/crio-e550a3c6d5b5c27ce08ebdefbce59afe8e9c6c07dd3205c7403d063525ac99dc WatchSource:0}: Error finding container e550a3c6d5b5c27ce08ebdefbce59afe8e9c6c07dd3205c7403d063525ac99dc: Status 404 returned error can't find the container with id e550a3c6d5b5c27ce08ebdefbce59afe8e9c6c07dd3205c7403d063525ac99dc
Sep 10 21:19:24.745599 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.745552    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 21:14:23 +0000 UTC" deadline="2028-03-18 19:34:33.090678661 +0000 UTC"
Sep 10 21:19:24.745599 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.745597    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13318h15m8.345085238s"
Sep 10 21:19:24.823403 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.823280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m2mjz" event={"ID":"1e48ee8f-64c3-4ab2-950c-f09d5043d472","Type":"ContainerStarted","Data":"cd11c8c23aa777ea90a37121b2f03f5c3708fae7b8a757ce3210500c962b90b6"}
Sep 10 21:19:24.825378 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.825343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9txsp" event={"ID":"eef3bddd-76e7-4f9f-867c-1af56e489709","Type":"ContainerStarted","Data":"10ea52a1bdf846c6bdd3f6fc701d7e42b03c58a20807a5b98770df6706913938"}
Sep 10 21:19:24.826754 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.826728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln" event={"ID":"e8581684-62f8-4d33-bce5-4e3a1e7a25e4","Type":"ContainerStarted","Data":"e550a3c6d5b5c27ce08ebdefbce59afe8e9c6c07dd3205c7403d063525ac99dc"}
Sep 10 21:19:24.828226 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.828171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zs4b5" event={"ID":"6d2f5710-7e64-49ab-a1c1-6d5cce892e5e","Type":"ContainerStarted","Data":"c036ac63b4bbd575037cb7d70ecc922dda70bd8e35540f03556b35ffbb4ed175"}
Sep 10 21:19:24.829424 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.829383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxt6" event={"ID":"ed0bb520-aceb-44a9-8796-178e661d1482","Type":"ContainerStarted","Data":"45e2f5f68a7558e822cb3f83df79bb729359d1a1c0c155d09a6c633a13c01862"}
Sep 10 21:19:24.830632 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.830612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zzrvw" event={"ID":"6987485a-cc5d-45a1-a8e4-1eb4cf948766","Type":"ContainerStarted","Data":"6f8f62c3124c925eaad031b2d0fef92266d5f3fbb2f7206978269b0526415120"}
Sep 10 21:19:24.832784 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.832759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal" event={"ID":"2d6ceb36f4f0ad1940c1c73fe62eb8fd","Type":"ContainerStarted","Data":"dedb59fdd18a42bcc1570e3aca5bc902706d11af0eecfe6fb4f602b9961f8db3"}
Sep 10 21:19:24.835287 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.835254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m" event={"ID":"6e22316c-8ace-43e8-a93f-2e7272222b9c","Type":"ContainerStarted","Data":"61fe727fc8420254baae083ad0c2ac27fdf44b1c14b29ee2cb769d86677e7e6b"}
Sep 10 21:19:24.839197 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.839157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hxssh" event={"ID":"3a3eda3e-fd65-4cab-b5f2-2870340b88b4","Type":"ContainerStarted","Data":"753637f87299f0303555cf875c0946677fb2fca81baa1ce22030c5983f342245"}
Sep 10 21:19:24.840316 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.840297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lj6vm" event={"ID":"3f067fda-a406-4cf0-9135-b1efa82f373f","Type":"ContainerStarted","Data":"41c545e133e289ab8c31a8385dd1df3f72fc574444958633cda604cd455c014c"}
Sep 10 21:19:24.848775 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:24.848706    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal" podStartSLOduration=1.848694198 podStartE2EDuration="1.848694198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:24.848172971 +0000 UTC m=+3.496087138" watchObservedRunningTime="2026-09-10 21:19:24.848694198 +0000 UTC m=+3.496608365"
Sep 10 21:19:25.331012 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:25.330967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:25.331218 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:25.331129    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:25.331218 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:25.331194    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs podName:51decde2-f81d-4c6f-b857-8fd6150f288a nodeName:}" failed. No retries permitted until 2026-09-10 21:19:27.331174945 +0000 UTC m=+5.979089095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs") pod "network-metrics-daemon-kgjnr" (UID: "51decde2-f81d-4c6f-b857-8fd6150f288a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:25.432556 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:25.432154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:25.432556 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:25.432218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbh2\" (UniqueName: \"kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2\") pod \"network-check-target-dz7bd\" (UID: \"9dca27a7-8d1a-4c60-bacf-bfad74056541\") " pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:25.432556 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:25.432334    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:25.432556 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:25.432377    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 21:19:25.432556 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:25.432397    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 21:19:25.432556 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:25.432426    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rxbh2 for pod openshift-network-diagnostics/network-check-target-dz7bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:25.432556 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:25.432439    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret podName:7e804440-819b-402d-ba84-f9f0c3ad4aeb nodeName:}" failed. No retries permitted until 2026-09-10 21:19:27.432399782 +0000 UTC m=+6.080313947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret") pod "global-pull-secret-syncer-2r6cp" (UID: "7e804440-819b-402d-ba84-f9f0c3ad4aeb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:25.432556 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:25.432478    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2 podName:9dca27a7-8d1a-4c60-bacf-bfad74056541 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:27.432462371 +0000 UTC m=+6.080376521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rxbh2" (UniqueName: "kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2") pod "network-check-target-dz7bd" (UID: "9dca27a7-8d1a-4c60-bacf-bfad74056541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:25.746883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:25.746755    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 21:14:23 +0000 UTC" deadline="2028-05-10 16:26:25.192834844 +0000 UTC"
Sep 10 21:19:25.746883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:25.746816    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14587h6m59.446023024s"
Sep 10 21:19:25.817304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:25.817274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:25.817488 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:25.817428    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:25.817872 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:25.817845    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:25.817982 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:25.817949    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:25.818047 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:25.818014    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:25.818098 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:25.818086    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:25.869175 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:25.867766    2580 generic.go:356] "Generic (PLEG): container finished" podID="30a08eec8551a0009e61c657650503a2" containerID="3847f6caac7694217f2cb81a669438b32c02936d2ad44199d37cef50f3318395" exitCode=0
Sep 10 21:19:25.869175 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:25.868938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal" event={"ID":"30a08eec8551a0009e61c657650503a2","Type":"ContainerDied","Data":"3847f6caac7694217f2cb81a669438b32c02936d2ad44199d37cef50f3318395"}
Sep 10 21:19:26.895698 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:26.895659    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal" event={"ID":"30a08eec8551a0009e61c657650503a2","Type":"ContainerStarted","Data":"f1afdd8e1eea667bc11e152848f087266563441f77636b8e05c07ef75a18d3bb"}
Sep 10 21:19:27.346592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:27.346499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:27.346760 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:27.346672    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:27.346760 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:27.346744    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs podName:51decde2-f81d-4c6f-b857-8fd6150f288a nodeName:}" failed. No retries permitted until 2026-09-10 21:19:31.346720293 +0000 UTC m=+9.994634463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs") pod "network-metrics-daemon-kgjnr" (UID: "51decde2-f81d-4c6f-b857-8fd6150f288a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:27.447289 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:27.447250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:27.447504 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:27.447316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbh2\" (UniqueName: \"kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2\") pod \"network-check-target-dz7bd\" (UID: \"9dca27a7-8d1a-4c60-bacf-bfad74056541\") " pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:27.447574 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:27.447492    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 21:19:27.447574 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:27.447527    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 21:19:27.447574 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:27.447540    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rxbh2 for pod openshift-network-diagnostics/network-check-target-dz7bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:27.447731 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:27.447599    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2 podName:9dca27a7-8d1a-4c60-bacf-bfad74056541 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:31.447579476 +0000 UTC m=+10.095493628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rxbh2" (UniqueName: "kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2") pod "network-check-target-dz7bd" (UID: "9dca27a7-8d1a-4c60-bacf-bfad74056541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:27.448030 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:27.448008    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:27.448107 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:27.448064    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret podName:7e804440-819b-402d-ba84-f9f0c3ad4aeb nodeName:}" failed. No retries permitted until 2026-09-10 21:19:31.448050011 +0000 UTC m=+10.095964173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret") pod "global-pull-secret-syncer-2r6cp" (UID: "7e804440-819b-402d-ba84-f9f0c3ad4aeb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:27.815627 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:27.815517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:27.815627 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:27.815552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:27.815856 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:27.815682    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:27.815856 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:27.815805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:27.815947 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:27.815901    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:27.816041 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:27.815992    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:29.814242 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:29.814206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:29.814710 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:29.814206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:29.814710 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:29.814368    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:29.814710 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:29.814426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:29.814710 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:29.814532    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:29.814710 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:29.814620    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:31.381878 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:31.381836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:31.382341 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:31.381960    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:31.382341 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:31.382041    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs podName:51decde2-f81d-4c6f-b857-8fd6150f288a nodeName:}" failed. No retries permitted until 2026-09-10 21:19:39.382017434 +0000 UTC m=+18.029931587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs") pod "network-metrics-daemon-kgjnr" (UID: "51decde2-f81d-4c6f-b857-8fd6150f288a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:31.482928 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:31.482882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbh2\" (UniqueName: \"kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2\") pod \"network-check-target-dz7bd\" (UID: \"9dca27a7-8d1a-4c60-bacf-bfad74056541\") " pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:31.483098 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:31.482973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:31.483098 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:31.483067    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:31.483098 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:31.483078    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 21:19:31.483201 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:31.483105    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 21:19:31.483201 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:31.483118    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rxbh2 for pod openshift-network-diagnostics/network-check-target-dz7bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:31.483201 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:31.483122    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret podName:7e804440-819b-402d-ba84-f9f0c3ad4aeb nodeName:}" failed. No retries permitted until 2026-09-10 21:19:39.483108344 +0000 UTC m=+18.131022502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret") pod "global-pull-secret-syncer-2r6cp" (UID: "7e804440-819b-402d-ba84-f9f0c3ad4aeb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:31.483201 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:31.483166    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2 podName:9dca27a7-8d1a-4c60-bacf-bfad74056541 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:39.483153757 +0000 UTC m=+18.131067904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rxbh2" (UniqueName: "kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2") pod "network-check-target-dz7bd" (UID: "9dca27a7-8d1a-4c60-bacf-bfad74056541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:31.815378 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:31.815287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:31.815564 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:31.815389    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:31.815976 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:31.815743    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:31.815976 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:31.815838    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:31.815976 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:31.815879    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:31.815976 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:31.815949    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:33.814899 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:33.814855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:33.814899 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:33.814886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:33.815351 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:33.814869    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:33.815351 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:33.815022    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:33.815351 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:33.815144    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:33.815351 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:33.815290    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:35.814331 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:35.814294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:35.814850 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:35.814294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:35.814850 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:35.814455    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:35.814850 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:35.814499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:35.814850 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:35.814586    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:35.814850 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:35.814651    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:37.814350 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:37.814318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:37.814350 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:37.814346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:37.814842 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:37.814329    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:37.814842 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:37.814473    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:37.814842 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:37.814574    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:37.814842 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:37.814657    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:39.444739 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:39.444702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:39.445158 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:39.444852    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:39.445158 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:39.444931    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs podName:51decde2-f81d-4c6f-b857-8fd6150f288a nodeName:}" failed. No retries permitted until 2026-09-10 21:19:55.444910591 +0000 UTC m=+34.092824756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs") pod "network-metrics-daemon-kgjnr" (UID: "51decde2-f81d-4c6f-b857-8fd6150f288a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:39.545833 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:39.545802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:39.546061 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:39.545858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbh2\" (UniqueName: \"kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2\") pod \"network-check-target-dz7bd\" (UID: \"9dca27a7-8d1a-4c60-bacf-bfad74056541\") " pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:39.546061 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:39.545963    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:39.546061 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:39.545975    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 21:19:39.546061 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:39.545995    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 21:19:39.546061 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:39.546008    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rxbh2 for pod openshift-network-diagnostics/network-check-target-dz7bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:39.546061 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:39.546035    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret podName:7e804440-819b-402d-ba84-f9f0c3ad4aeb nodeName:}" failed. No retries permitted until 2026-09-10 21:19:55.546015648 +0000 UTC m=+34.193929813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret") pod "global-pull-secret-syncer-2r6cp" (UID: "7e804440-819b-402d-ba84-f9f0c3ad4aeb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:39.546061 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:39.546058    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2 podName:9dca27a7-8d1a-4c60-bacf-bfad74056541 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:55.546044681 +0000 UTC m=+34.193958846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rxbh2" (UniqueName: "kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2") pod "network-check-target-dz7bd" (UID: "9dca27a7-8d1a-4c60-bacf-bfad74056541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:39.814877 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:39.814797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:39.814877 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:39.814834    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:39.815077 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:39.814798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:39.815077 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:39.814933    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:39.815077 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:39.815040    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:39.815190 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:39.815115    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:41.815352 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:41.815302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:41.815839 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:41.815435    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:41.815839 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:41.815462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:41.815839 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:41.815509    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:41.815839 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:41.815531    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:41.815839 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:41.815622    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:42.930255 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.930199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9txsp" event={"ID":"eef3bddd-76e7-4f9f-867c-1af56e489709","Type":"ContainerStarted","Data":"e2155cae32101ece4dcf7d002ea0afad10aeee317285305a1d808922c18e1c73"}
Sep 10 21:19:42.931717 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.931688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln" event={"ID":"e8581684-62f8-4d33-bce5-4e3a1e7a25e4","Type":"ContainerStarted","Data":"49a3480f5e8931990d39f39f23b517bdbb1a0eb4b1c11fdac7e5be143030e8e1"}
Sep 10 21:19:42.932960 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.932929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zs4b5" event={"ID":"6d2f5710-7e64-49ab-a1c1-6d5cce892e5e","Type":"ContainerStarted","Data":"ea7b3a2126eead7f27220c5988f06a7fc22c74261bbccfb3b63921ecb90d4c8f"}
Sep 10 21:19:42.934757 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.934734    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed0bb520-aceb-44a9-8796-178e661d1482" containerID="9e064864b278c67d174debb8187bbbc9d8508d853266cea46586035c2a864ca0" exitCode=0
Sep 10 21:19:42.934858 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.934800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxt6" event={"ID":"ed0bb520-aceb-44a9-8796-178e661d1482","Type":"ContainerDied","Data":"9e064864b278c67d174debb8187bbbc9d8508d853266cea46586035c2a864ca0"}
Sep 10 21:19:42.936318 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.936296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zzrvw" event={"ID":"6987485a-cc5d-45a1-a8e4-1eb4cf948766","Type":"ContainerStarted","Data":"0a211f14acff69ce05c9a50d73064da09704d241bfdc04fe6ca7a8c0770a3ecd"}
Sep 10 21:19:42.938952 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.938930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m" event={"ID":"6e22316c-8ace-43e8-a93f-2e7272222b9c","Type":"ContainerStarted","Data":"c97f6197a96a206ece47659beea585e04911588c7431c22433cd71324d6f079e"}
Sep 10 21:19:42.939047 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.938994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m" event={"ID":"6e22316c-8ace-43e8-a93f-2e7272222b9c","Type":"ContainerStarted","Data":"0378bcb922cc50ab03ae464450b1efcc4867d5f2467d49836c942151722b490d"}
Sep 10 21:19:42.939047 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.939034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m" event={"ID":"6e22316c-8ace-43e8-a93f-2e7272222b9c","Type":"ContainerStarted","Data":"2c7145fd9f741a23f401ee79fa507fb71022443a6904fd48922ced8ac84c7265"}
Sep 10 21:19:42.939149 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.939074    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m" event={"ID":"6e22316c-8ace-43e8-a93f-2e7272222b9c","Type":"ContainerStarted","Data":"97c2ba4ddbc507304610365fc7425f68d5f9d7402f25c10922f79627363c1936"}
Sep 10 21:19:42.940338 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.940316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lj6vm" event={"ID":"3f067fda-a406-4cf0-9135-b1efa82f373f","Type":"ContainerStarted","Data":"7d6e9a085f9d72a5de26ba4d498d6d9fc4281238627dc0d7250f0fb0452273eb"}
Sep 10 21:19:42.941761 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.941737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m2mjz" event={"ID":"1e48ee8f-64c3-4ab2-950c-f09d5043d472","Type":"ContainerStarted","Data":"734cbd3d4409a977f07443ab5f3e1624c61f2f2c97237dfb2e015dbe7e96ac5d"}
Sep 10 21:19:42.948126 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.948091    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal" podStartSLOduration=19.948062508 podStartE2EDuration="19.948062508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:26.912654804 +0000 UTC m=+5.560568972" watchObservedRunningTime="2026-09-10 21:19:42.948062508 +0000 UTC m=+21.595976668"
Sep 10 21:19:42.948589 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.948565    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9txsp" podStartSLOduration=4.098193875 podStartE2EDuration="21.948556572s" totalImagesPullingTime="17.850362697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:42.948028852 +0000 UTC m=+21.595943019" watchObservedRunningTime="2026-09-10 21:19:42.948556572 +0000 UTC m=+21.596470740"
Sep 10 21:19:42.962018 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.961983    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lj6vm" podStartSLOduration=4.12882193 podStartE2EDuration="21.961972261s" totalImagesPullingTime="17.833150331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:42.961822585 +0000 UTC m=+21.609736754" watchObservedRunningTime="2026-09-10 21:19:42.961972261 +0000 UTC m=+21.609886429"
Sep 10 21:19:42.981107 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:42.981073    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m2mjz" podStartSLOduration=4.094672689 podStartE2EDuration="21.98106237s" totalImagesPullingTime="17.886389681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:42.98073012 +0000 UTC m=+21.628644289" watchObservedRunningTime="2026-09-10 21:19:42.98106237 +0000 UTC m=+21.628976537"
Sep 10 21:19:43.012943 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:43.012909    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zzrvw" podStartSLOduration=3.182470257 podStartE2EDuration="21.012895921s" totalImagesPullingTime="17.830425664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:43.012709911 +0000 UTC m=+21.660624078" watchObservedRunningTime="2026-09-10 21:19:43.012895921 +0000 UTC m=+21.660810088"
Sep 10 21:19:43.025795 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:43.025760    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zs4b5" podStartSLOduration=7.558642503 podStartE2EDuration="22.025747638s" totalImagesPullingTime="14.467105135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:43.02570748 +0000 UTC m=+21.673621648" watchObservedRunningTime="2026-09-10 21:19:43.025747638 +0000 UTC m=+21.673661806"
Sep 10 21:19:43.814104 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:43.814073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:43.814315 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:43.814190    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:43.814664 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:43.814630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:43.814805 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:43.814732    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:43.814872 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:43.814821    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:43.814917 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:43.814895    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:43.949208 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:43.949172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m" event={"ID":"6e22316c-8ace-43e8-a93f-2e7272222b9c","Type":"ContainerStarted","Data":"0c19a52702a71c67db7cdfa407a0dcc5c1023401cb2b89d63b671571e03452f1"}
Sep 10 21:19:43.949724 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:43.949241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m" event={"ID":"6e22316c-8ace-43e8-a93f-2e7272222b9c","Type":"ContainerStarted","Data":"5ae5dc5e9c8f3bbca3548919f05e66d7a28cb4e42246007ef31bde2808eaa532"}
Sep 10 21:19:43.952095 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:43.952073    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hxssh" event={"ID":"3a3eda3e-fd65-4cab-b5f2-2870340b88b4","Type":"ContainerStarted","Data":"a65cd83b53ae2d4051e95fc91c1b94f31c006bfb84fdfe714f6c94fb1e424ac5"}
Sep 10 21:19:43.967437 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:43.967377    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hxssh" podStartSLOduration=5.135455357 podStartE2EDuration="22.967352578s" totalImagesPullingTime="17.831897221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:43.967012824 +0000 UTC m=+22.614926993" watchObservedRunningTime="2026-09-10 21:19:43.967352578 +0000 UTC m=+22.615266770"
Sep 10 21:19:44.445386 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:44.445163    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 21:19:44.572304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:44.572195    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:44.572888 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:44.572867    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:44.782186 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:44.782079    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T21:19:44.445365597Z","UUID":"2aa510db-5616-4340-b36f-458b6fdf07e6","Handler":null,"Name":"","Endpoint":""}
Sep 10 21:19:44.783909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:44.783887    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 21:19:44.784040 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:44.783925    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 21:19:44.956038 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:44.955969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln" event={"ID":"e8581684-62f8-4d33-bce5-4e3a1e7a25e4","Type":"ContainerStarted","Data":"9a72eef79aa1dd0dbdbae32c6cca7c3140b822ff19e0c8b2ad7f98b592c31a06"}
Sep 10 21:19:44.956485 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:44.956364    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:44.956861 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:44.956840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zs4b5"
Sep 10 21:19:45.817993 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:45.817966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:45.818135 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:45.818069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:45.818135 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:45.818077    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:45.818135 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:45.818085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:45.818307 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:45.818177    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:45.818307 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:45.818275    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:45.962163 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:45.962117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln" event={"ID":"e8581684-62f8-4d33-bce5-4e3a1e7a25e4","Type":"ContainerStarted","Data":"957157b86f746113d035def18c21d32302fedf99adf36c33b406c64b60e45cc8"}
Sep 10 21:19:45.966334 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:45.966286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m" event={"ID":"6e22316c-8ace-43e8-a93f-2e7272222b9c","Type":"ContainerStarted","Data":"8f37d18aba9ed6a557bf449c411d63b6b205adf90de297b4883b38f16ce094df"}
Sep 10 21:19:45.979818 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:45.979760    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-829ln" podStartSLOduration=4.1841005 podStartE2EDuration="24.979743329s" totalImagesPullingTime="20.795642829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:45.979718013 +0000 UTC m=+24.627632182" watchObservedRunningTime="2026-09-10 21:19:45.979743329 +0000 UTC m=+24.627657532"
Sep 10 21:19:47.817769 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:47.817471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:47.817769 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:47.817498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:47.818106 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:47.817775    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:47.818106 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:47.817888    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:47.818106 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:47.817521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:47.818106 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:47.817991    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:47.973388 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:47.973341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxt6" event={"ID":"ed0bb520-aceb-44a9-8796-178e661d1482","Type":"ContainerStarted","Data":"6b01704994ef911864fb03a794d89e6907e9b81f0983d8e1b434a6fda7af3c5c"}
Sep 10 21:19:47.977501 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:47.977474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m" event={"ID":"6e22316c-8ace-43e8-a93f-2e7272222b9c","Type":"ContainerStarted","Data":"45eb2c4b24d6fe58fc8f364cdb6022cf1d99aa6daa7f161ff7b7e3f41180726c"}
Sep 10 21:19:47.977791 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:47.977778    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:47.993269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:47.993241    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:48.028451 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:48.028387    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m" podStartSLOduration=9.124468711 podStartE2EDuration="27.028371797s" totalImagesPullingTime="17.903903086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:48.027943589 +0000 UTC m=+26.675857771" watchObservedRunningTime="2026-09-10 21:19:48.028371797 +0000 UTC m=+26.676285964"
Sep 10 21:19:48.982000 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:48.981970    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed0bb520-aceb-44a9-8796-178e661d1482" containerID="6b01704994ef911864fb03a794d89e6907e9b81f0983d8e1b434a6fda7af3c5c" exitCode=0
Sep 10 21:19:48.982619 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:48.982050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxt6" event={"ID":"ed0bb520-aceb-44a9-8796-178e661d1482","Type":"ContainerDied","Data":"6b01704994ef911864fb03a794d89e6907e9b81f0983d8e1b434a6fda7af3c5c"}
Sep 10 21:19:48.982619 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:48.982532    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:48.982619 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:48.982610    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:48.997934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:48.997910    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:19:49.670701 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:49.670666    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2r6cp"]
Sep 10 21:19:49.670864 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:49.670836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:49.670964 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:49.670943    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:49.674094 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:49.674065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kgjnr"]
Sep 10 21:19:49.674200 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:49.674175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:49.674323 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:49.674282    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:49.686637 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:49.686576    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dz7bd"]
Sep 10 21:19:49.686747 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:49.686667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:49.686747 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:49.686730    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:50.814620 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:50.814433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:50.815145 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:50.814688    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:50.988379 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:50.988343    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed0bb520-aceb-44a9-8796-178e661d1482" containerID="f702f00edca95c8f6ed367e26dbc3fc0dc8f661e5512c394a668f878006c8a31" exitCode=0
Sep 10 21:19:50.988548 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:50.988405    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxt6" event={"ID":"ed0bb520-aceb-44a9-8796-178e661d1482","Type":"ContainerDied","Data":"f702f00edca95c8f6ed367e26dbc3fc0dc8f661e5512c394a668f878006c8a31"}
Sep 10 21:19:51.815171 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:51.815148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:51.815554 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:51.815230    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:51.815554 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:51.815251    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:51.815554 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:51.815351    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:51.993388 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:51.993306    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed0bb520-aceb-44a9-8796-178e661d1482" containerID="41b96e1b16f84696f13cd559397b8b8d0af130f38403654c85c1938d06bc75d0" exitCode=0
Sep 10 21:19:51.993388 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:51.993376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxt6" event={"ID":"ed0bb520-aceb-44a9-8796-178e661d1482","Type":"ContainerDied","Data":"41b96e1b16f84696f13cd559397b8b8d0af130f38403654c85c1938d06bc75d0"}
Sep 10 21:19:52.814810 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:52.814775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:52.814977 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:52.814899    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:53.814434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:53.814381    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:53.814879 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:53.814521    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2r6cp" podUID="7e804440-819b-402d-ba84-f9f0c3ad4aeb"
Sep 10 21:19:53.814879 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:53.814603    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:53.814879 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:53.814706    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kgjnr" podUID="51decde2-f81d-4c6f-b857-8fd6150f288a"
Sep 10 21:19:54.814756 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:54.814728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:54.815120 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:54.814827    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz7bd" podUID="9dca27a7-8d1a-4c60-bacf-bfad74056541"
Sep 10 21:19:55.169333 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.169265    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeReady"
Sep 10 21:19:55.169512 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.169433    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 21:19:55.202957 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.202927    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-685bcdb8f9-75qqc"]
Sep 10 21:19:55.230169 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.229372    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s8nr2"]
Sep 10 21:19:55.230169 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.229602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.232727 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.232701    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 21:19:55.233023 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.233002    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 21:19:55.233023 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.233014    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mvsp2\""
Sep 10 21:19:55.233252 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.233050    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 21:19:55.238229 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.238213    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 21:19:55.242587 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.242570    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qccrj"]
Sep 10 21:19:55.242697 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.242681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.244971 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.244950    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 21:19:55.245072 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.244957    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f6g5g\""
Sep 10 21:19:55.245135 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.245128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 21:19:55.264925 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.264898    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-685bcdb8f9-75qqc"]
Sep 10 21:19:55.265003 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.264934    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s8nr2"]
Sep 10 21:19:55.265003 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.264962    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qccrj"]
Sep 10 21:19:55.265085 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.265006    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qccrj"
Sep 10 21:19:55.267844 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.267698    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 21:19:55.267844 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.267807    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 21:19:55.267844 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.267824    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 21:19:55.268041 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.267960    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bgqsb\""
Sep 10 21:19:55.361660 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.361624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.361660 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.361662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-trusted-ca\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.361880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.361678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-bound-sa-token\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.361880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.361708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9abb731-3053-4f97-ba96-f2b19f48bd3c-cert\") pod \"ingress-canary-qccrj\" (UID: \"e9abb731-3053-4f97-ba96-f2b19f48bd3c\") " pod="openshift-ingress-canary/ingress-canary-qccrj"
Sep 10 21:19:55.361880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.361731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrvwv\" (UniqueName: \"kubernetes.io/projected/e9abb731-3053-4f97-ba96-f2b19f48bd3c-kube-api-access-hrvwv\") pod \"ingress-canary-qccrj\" (UID: \"e9abb731-3053-4f97-ba96-f2b19f48bd3c\") " pod="openshift-ingress-canary/ingress-canary-qccrj"
Sep 10 21:19:55.361880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.361775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f48024dc-9b26-46bf-8da8-b9ef5383bf45-config-volume\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.361880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.361819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-certificates\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.361880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.361847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.361880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.361864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48024dc-9b26-46bf-8da8-b9ef5383bf45-tmp-dir\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.362215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.361895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5gd\" (UniqueName: \"kubernetes.io/projected/f48024dc-9b26-46bf-8da8-b9ef5383bf45-kube-api-access-xg5gd\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.362215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.361942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebabd368-7bcb-4047-9e93-88072e7887a4-ca-trust-extracted\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.362215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.362035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-image-registry-private-configuration\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.362215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.362097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbmhh\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-kube-api-access-fbmhh\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.362215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.362132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-installation-pull-secrets\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.463568 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.463530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.463729 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.463576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-trusted-ca\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.463729 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.463602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-bound-sa-token\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.463729 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.463629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9abb731-3053-4f97-ba96-f2b19f48bd3c-cert\") pod \"ingress-canary-qccrj\" (UID: \"e9abb731-3053-4f97-ba96-f2b19f48bd3c\") " pod="openshift-ingress-canary/ingress-canary-qccrj"
Sep 10 21:19:55.463729 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.463653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrvwv\" (UniqueName: \"kubernetes.io/projected/e9abb731-3053-4f97-ba96-f2b19f48bd3c-kube-api-access-hrvwv\") pod \"ingress-canary-qccrj\" (UID: \"e9abb731-3053-4f97-ba96-f2b19f48bd3c\") " pod="openshift-ingress-canary/ingress-canary-qccrj"
Sep 10 21:19:55.463729 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.463668    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 21:19:55.463729 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.463692    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-685bcdb8f9-75qqc: secret "image-registry-tls" not found
Sep 10 21:19:55.464035 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.463737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f48024dc-9b26-46bf-8da8-b9ef5383bf45-config-volume\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.464035 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.463768    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls podName:ebabd368-7bcb-4047-9e93-88072e7887a4 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:55.963744283 +0000 UTC m=+34.611658454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls") pod "image-registry-685bcdb8f9-75qqc" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4") : secret "image-registry-tls" not found
Sep 10 21:19:55.464035 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.463823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-certificates\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.464035 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.463867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.464035 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.463891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48024dc-9b26-46bf-8da8-b9ef5383bf45-tmp-dir\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.464035 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.463930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5gd\" (UniqueName: \"kubernetes.io/projected/f48024dc-9b26-46bf-8da8-b9ef5383bf45-kube-api-access-xg5gd\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.464035 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.463955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebabd368-7bcb-4047-9e93-88072e7887a4-ca-trust-extracted\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.464035 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.464008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:55.464035 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.464036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-image-registry-private-configuration\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.464497 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.464077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbmhh\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-kube-api-access-fbmhh\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.464497 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.464105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-installation-pull-secrets\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.464497 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.464214    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:55.464497 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.464285    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs podName:51decde2-f81d-4c6f-b857-8fd6150f288a nodeName:}" failed. No retries permitted until 2026-09-10 21:20:27.464266071 +0000 UTC m=+66.112180240 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs") pod "network-metrics-daemon-kgjnr" (UID: "51decde2-f81d-4c6f-b857-8fd6150f288a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 21:19:55.464497 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.464309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48024dc-9b26-46bf-8da8-b9ef5383bf45-tmp-dir\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.464497 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.464335    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 21:19:55.464497 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.464347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f48024dc-9b26-46bf-8da8-b9ef5383bf45-config-volume\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.464497 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.464375    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls podName:f48024dc-9b26-46bf-8da8-b9ef5383bf45 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:55.964362797 +0000 UTC m=+34.612276944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls") pod "dns-default-s8nr2" (UID: "f48024dc-9b26-46bf-8da8-b9ef5383bf45") : secret "dns-default-metrics-tls" not found
Sep 10 21:19:55.464858 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.464503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-certificates\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.464858 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.464688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-trusted-ca\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.464858 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.464803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebabd368-7bcb-4047-9e93-88072e7887a4-ca-trust-extracted\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.468452 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.468405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-installation-pull-secrets\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.468558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.468457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9abb731-3053-4f97-ba96-f2b19f48bd3c-cert\") pod \"ingress-canary-qccrj\" (UID: \"e9abb731-3053-4f97-ba96-f2b19f48bd3c\") " pod="openshift-ingress-canary/ingress-canary-qccrj"
Sep 10 21:19:55.474880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.474206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrvwv\" (UniqueName: \"kubernetes.io/projected/e9abb731-3053-4f97-ba96-f2b19f48bd3c-kube-api-access-hrvwv\") pod \"ingress-canary-qccrj\" (UID: \"e9abb731-3053-4f97-ba96-f2b19f48bd3c\") " pod="openshift-ingress-canary/ingress-canary-qccrj"
Sep 10 21:19:55.474880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.474495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5gd\" (UniqueName: \"kubernetes.io/projected/f48024dc-9b26-46bf-8da8-b9ef5383bf45-kube-api-access-xg5gd\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.474880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.474854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-bound-sa-token\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.475116 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.475097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbmhh\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-kube-api-access-fbmhh\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.478103 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.478085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-image-registry-private-configuration\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.565273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.565068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:55.565273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.565188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbh2\" (UniqueName: \"kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2\") pod \"network-check-target-dz7bd\" (UID: \"9dca27a7-8d1a-4c60-bacf-bfad74056541\") " pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:55.568357 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.565628    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:55.568357 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.565727    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 21:19:55.568357 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.565760    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 21:19:55.568357 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.565777    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rxbh2 for pod openshift-network-diagnostics/network-check-target-dz7bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:55.568357 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.565733    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret podName:7e804440-819b-402d-ba84-f9f0c3ad4aeb nodeName:}" failed. No retries permitted until 2026-09-10 21:20:27.565703586 +0000 UTC m=+66.213617749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret") pod "global-pull-secret-syncer-2r6cp" (UID: "7e804440-819b-402d-ba84-f9f0c3ad4aeb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 21:19:55.568357 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.565851    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2 podName:9dca27a7-8d1a-4c60-bacf-bfad74056541 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:27.565834274 +0000 UTC m=+66.213748420 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rxbh2" (UniqueName: "kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2") pod "network-check-target-dz7bd" (UID: "9dca27a7-8d1a-4c60-bacf-bfad74056541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 21:19:55.574769 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.574735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qccrj"
Sep 10 21:19:55.814090 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.813999    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:19:55.814430 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.814395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:19:55.817731 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.817709    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c8fbc\""
Sep 10 21:19:55.818064 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.817777    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 21:19:55.818126 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.818110    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 21:19:55.968989 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.968943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:55.969167 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:55.969051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:55.969227 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.969166    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 21:19:55.969227 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.969219    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls podName:f48024dc-9b26-46bf-8da8-b9ef5383bf45 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:56.969205413 +0000 UTC m=+35.617119573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls") pod "dns-default-s8nr2" (UID: "f48024dc-9b26-46bf-8da8-b9ef5383bf45") : secret "dns-default-metrics-tls" not found
Sep 10 21:19:55.969670 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.969650    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 21:19:55.969670 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.969671    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-685bcdb8f9-75qqc: secret "image-registry-tls" not found
Sep 10 21:19:55.969812 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:55.969716    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls podName:ebabd368-7bcb-4047-9e93-88072e7887a4 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:56.969700365 +0000 UTC m=+35.617614514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls") pod "image-registry-685bcdb8f9-75qqc" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4") : secret "image-registry-tls" not found
Sep 10 21:19:56.814857 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:56.814789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:19:56.817174 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:56.817147    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 21:19:56.817174 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:56.817147    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 21:19:56.817372 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:56.817249    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kc896\""
Sep 10 21:19:56.821822 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:56.821804    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qccrj"]
Sep 10 21:19:56.825483 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:56.825459    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9abb731_3053_4f97_ba96_f2b19f48bd3c.slice/crio-3706a927331300da638746fa05c26eae3f12353ca335e34ad25e097133d9e7ff WatchSource:0}: Error finding container 3706a927331300da638746fa05c26eae3f12353ca335e34ad25e097133d9e7ff: Status 404 returned error can't find the container with id 3706a927331300da638746fa05c26eae3f12353ca335e34ad25e097133d9e7ff
Sep 10 21:19:56.976694 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:56.976664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:56.976767 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:56.976741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:56.976865 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:56.976796    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 21:19:56.976915 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:56.976858    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 21:19:56.976915 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:56.976875    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-685bcdb8f9-75qqc: secret "image-registry-tls" not found
Sep 10 21:19:56.976915 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:56.976866    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls podName:f48024dc-9b26-46bf-8da8-b9ef5383bf45 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:58.976845661 +0000 UTC m=+37.624759823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls") pod "dns-default-s8nr2" (UID: "f48024dc-9b26-46bf-8da8-b9ef5383bf45") : secret "dns-default-metrics-tls" not found
Sep 10 21:19:56.977053 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:56.976930    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls podName:ebabd368-7bcb-4047-9e93-88072e7887a4 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:58.976915121 +0000 UTC m=+37.624829274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls") pod "image-registry-685bcdb8f9-75qqc" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4") : secret "image-registry-tls" not found
Sep 10 21:19:57.007913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.007876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxt6" event={"ID":"ed0bb520-aceb-44a9-8796-178e661d1482","Type":"ContainerStarted","Data":"d48791dbb350e78e39cd6c85d20e617f37bb064e54b9df52c78ed2c0106ff3a1"}
Sep 10 21:19:57.009131 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.009106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qccrj" event={"ID":"e9abb731-3053-4f97-ba96-f2b19f48bd3c","Type":"ContainerStarted","Data":"3706a927331300da638746fa05c26eae3f12353ca335e34ad25e097133d9e7ff"}
Sep 10 21:19:57.092464 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.092434    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2zz2b"]
Sep 10 21:19:57.114078 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.114052    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2zz2b"]
Sep 10 21:19:57.114196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.114134    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.116306 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.116290    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 21:19:57.116486 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.116468    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 21:19:57.116559 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.116470    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 21:19:57.116559 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.116522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jf9vp\""
Sep 10 21:19:57.116559 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.116535    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 21:19:57.178390 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.178369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-crio-socket\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.178507 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.178395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.178507 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.178489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-data-volume\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.178575 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.178517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.178575 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.178540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgcd8\" (UniqueName: \"kubernetes.io/projected/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-api-access-zgcd8\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.279815 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.279787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-crio-socket\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.279815 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.279818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.279952 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.279875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-data-volume\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.279952 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.279893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.279952 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.279912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgcd8\" (UniqueName: \"kubernetes.io/projected/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-api-access-zgcd8\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.280095 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.280013    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-crio-socket\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.280095 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:57.280031    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 21:19:57.280095 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:57.280036    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 21:19:57.280095 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:57.280089    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls podName:e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:57.780074662 +0000 UTC m=+36.427988808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2zz2b" (UID: "e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7") : secret "insights-runtime-extractor-tls" not found
Sep 10 21:19:57.280239 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:57.280119    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm podName:e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:57.780099763 +0000 UTC m=+36.428013936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2zz2b" (UID: "e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7") : configmap "kube-rbac-proxy" not found
Sep 10 21:19:57.280239 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.280223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-data-volume\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.288720 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.288701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgcd8\" (UniqueName: \"kubernetes.io/projected/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-api-access-zgcd8\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.525892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.525856    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762"]
Sep 10 21:19:57.542299 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.542268    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762"]
Sep 10 21:19:57.542465 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.542403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762"
Sep 10 21:19:57.544830 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.544808    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 21:19:57.544830 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.544822    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4jp8k\""
Sep 10 21:19:57.545119 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.545088    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 21:19:57.683469 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.683407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnnhz\" (UniqueName: \"kubernetes.io/projected/5fa64996-d18c-4fe3-b347-8da4b600a85b-kube-api-access-dnnhz\") pod \"migrator-8b4d697c5-km762\" (UID: \"5fa64996-d18c-4fe3-b347-8da4b600a85b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762"
Sep 10 21:19:57.784009 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.783919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.784009 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.783954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnnhz\" (UniqueName: \"kubernetes.io/projected/5fa64996-d18c-4fe3-b347-8da4b600a85b-kube-api-access-dnnhz\") pod \"migrator-8b4d697c5-km762\" (UID: \"5fa64996-d18c-4fe3-b347-8da4b600a85b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762"
Sep 10 21:19:57.784215 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:57.784091    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 21:19:57.784263 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.784204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:57.784263 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:57.784252    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm podName:e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:58.784225192 +0000 UTC m=+37.432139354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2zz2b" (UID: "e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7") : configmap "kube-rbac-proxy" not found
Sep 10 21:19:57.784349 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:57.784309    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 21:19:57.784397 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:57.784361    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls podName:e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7 nodeName:}" failed. No retries permitted until 2026-09-10 21:19:58.784345022 +0000 UTC m=+37.432259182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2zz2b" (UID: "e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7") : secret "insights-runtime-extractor-tls" not found
Sep 10 21:19:57.792525 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.792498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnnhz\" (UniqueName: \"kubernetes.io/projected/5fa64996-d18c-4fe3-b347-8da4b600a85b-kube-api-access-dnnhz\") pod \"migrator-8b4d697c5-km762\" (UID: \"5fa64996-d18c-4fe3-b347-8da4b600a85b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762"
Sep 10 21:19:57.853693 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:57.853654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762"
Sep 10 21:19:58.014985 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:58.014952    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed0bb520-aceb-44a9-8796-178e661d1482" containerID="d48791dbb350e78e39cd6c85d20e617f37bb064e54b9df52c78ed2c0106ff3a1" exitCode=0
Sep 10 21:19:58.015226 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:58.015010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxt6" event={"ID":"ed0bb520-aceb-44a9-8796-178e661d1482","Type":"ContainerDied","Data":"d48791dbb350e78e39cd6c85d20e617f37bb064e54b9df52c78ed2c0106ff3a1"}
Sep 10 21:19:58.142146 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:58.141716    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762"]
Sep 10 21:19:58.144729 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:19:58.144706    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa64996_d18c_4fe3_b347_8da4b600a85b.slice/crio-526b6e3adc7939a7a77ff6678d936352be14a4f038798b54c85b8454359c4b55 WatchSource:0}: Error finding container 526b6e3adc7939a7a77ff6678d936352be14a4f038798b54c85b8454359c4b55: Status 404 returned error can't find the container with id 526b6e3adc7939a7a77ff6678d936352be14a4f038798b54c85b8454359c4b55
Sep 10 21:19:58.794679 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:58.794604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:58.794814 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:58.794726    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 21:19:58.794814 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:58.794789    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm podName:e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:00.794769852 +0000 UTC m=+39.442684016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2zz2b" (UID: "e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7") : configmap "kube-rbac-proxy" not found
Sep 10 21:19:58.794814 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:58.794723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:19:58.794931 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:58.794818    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 21:19:58.794931 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:58.794874    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls podName:e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:00.794859259 +0000 UTC m=+39.442773410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2zz2b" (UID: "e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7") : secret "insights-runtime-extractor-tls" not found
Sep 10 21:19:58.839206 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:58.839177    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zzrvw_6987485a-cc5d-45a1-a8e4-1eb4cf948766/dns-node-resolver/0.log"
Sep 10 21:19:58.996994 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:58.996945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:19:58.997434 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:58.997124    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 21:19:58.997434 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:58.997143    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-685bcdb8f9-75qqc: secret "image-registry-tls" not found
Sep 10 21:19:58.997434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:58.997185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:19:58.997434 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:58.997208    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls podName:ebabd368-7bcb-4047-9e93-88072e7887a4 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:02.997189295 +0000 UTC m=+41.645103462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls") pod "image-registry-685bcdb8f9-75qqc" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4") : secret "image-registry-tls" not found
Sep 10 21:19:58.997434 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:58.997267    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 21:19:58.997434 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:19:58.997300    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls podName:f48024dc-9b26-46bf-8da8-b9ef5383bf45 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:02.99729197 +0000 UTC m=+41.645206137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls") pod "dns-default-s8nr2" (UID: "f48024dc-9b26-46bf-8da8-b9ef5383bf45") : secret "dns-default-metrics-tls" not found
Sep 10 21:19:59.021634 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:59.021608    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed0bb520-aceb-44a9-8796-178e661d1482" containerID="5c1c1406f55d9a62c2441b9c23d50b7bdb5ef8485d108d4d4acc3a2f94b16872" exitCode=0
Sep 10 21:19:59.021767 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:59.021680    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxt6" event={"ID":"ed0bb520-aceb-44a9-8796-178e661d1482","Type":"ContainerDied","Data":"5c1c1406f55d9a62c2441b9c23d50b7bdb5ef8485d108d4d4acc3a2f94b16872"}
Sep 10 21:19:59.023106 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:59.023054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762" event={"ID":"5fa64996-d18c-4fe3-b347-8da4b600a85b","Type":"ContainerStarted","Data":"526b6e3adc7939a7a77ff6678d936352be14a4f038798b54c85b8454359c4b55"}
Sep 10 21:19:59.439486 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:19:59.439462    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lj6vm_3f067fda-a406-4cf0-9135-b1efa82f373f/node-ca/0.log"
Sep 10 21:20:00.027252 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:00.027215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qccrj" event={"ID":"e9abb731-3053-4f97-ba96-f2b19f48bd3c","Type":"ContainerStarted","Data":"ca33f857554cd52d9a2ed6a0bf55b05e64e761c337f7039048fcadf651ed9a2f"}
Sep 10 21:20:00.030472 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:00.030445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxt6" event={"ID":"ed0bb520-aceb-44a9-8796-178e661d1482","Type":"ContainerStarted","Data":"aa33af3996ba31cb8db6188143e72fc743820a578021bec41a62024385b93bfa"}
Sep 10 21:20:00.045950 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:00.045909    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qccrj" podStartSLOduration=2.369461458 podStartE2EDuration="5.045895108s" totalImagesPullingTime="2.67643365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:00.044865038 +0000 UTC m=+38.692779206" watchObservedRunningTime="2026-09-10 21:20:00.045895108 +0000 UTC m=+38.693809276"
Sep 10 21:20:00.069145 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:00.069100    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cxxt6" podStartSLOduration=9.80169045 podStartE2EDuration="39.069086683s" totalImagesPullingTime="29.267396233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:00.068512587 +0000 UTC m=+38.716426755" watchObservedRunningTime="2026-09-10 21:20:00.069086683 +0000 UTC m=+38.717000850"
Sep 10 21:20:00.241059 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:00.241028    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qccrj_e9abb731-3053-4f97-ba96-f2b19f48bd3c/serve-healthcheck-canary/0.log"
Sep 10 21:20:00.811841 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:00.811643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:20:00.812039 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:00.811790    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 21:20:00.812039 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:00.811957    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 21:20:00.812039 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:00.811968    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls podName:e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:04.811949722 +0000 UTC m=+43.459863872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2zz2b" (UID: "e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7") : secret "insights-runtime-extractor-tls" not found
Sep 10 21:20:00.812039 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:00.811899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:20:00.812039 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:00.812007    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm podName:e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:04.811992695 +0000 UTC m=+43.459906865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2zz2b" (UID: "e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7") : configmap "kube-rbac-proxy" not found
Sep 10 21:20:01.035459 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:01.035404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762" event={"ID":"5fa64996-d18c-4fe3-b347-8da4b600a85b","Type":"ContainerStarted","Data":"32192085ef273029aa27aa695ac62250159fa9428a5c266fd571a193830a78c1"}
Sep 10 21:20:01.035909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:01.035529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762" event={"ID":"5fa64996-d18c-4fe3-b347-8da4b600a85b","Type":"ContainerStarted","Data":"2cd62544ca4f18df23ecf7b288e1f5c5d1702b95e7751a39c79eb9432baa47a3"}
Sep 10 21:20:01.052832 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:01.052781    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-km762" podStartSLOduration=1.788491097 podStartE2EDuration="4.052765215s" totalImagesPullingTime="2.264274118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:01.051708072 +0000 UTC m=+39.699622241" watchObservedRunningTime="2026-09-10 21:20:01.052765215 +0000 UTC m=+39.700679430"
Sep 10 21:20:03.030423 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:03.030372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:20:03.030784 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:03.030450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:20:03.030784 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:03.030533    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 21:20:03.030784 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:03.030545    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 21:20:03.030784 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:03.030558    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-685bcdb8f9-75qqc: secret "image-registry-tls" not found
Sep 10 21:20:03.030784 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:03.030599    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls podName:f48024dc-9b26-46bf-8da8-b9ef5383bf45 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:11.030582769 +0000 UTC m=+49.678496914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls") pod "dns-default-s8nr2" (UID: "f48024dc-9b26-46bf-8da8-b9ef5383bf45") : secret "dns-default-metrics-tls" not found
Sep 10 21:20:03.030784 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:03.030613    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls podName:ebabd368-7bcb-4047-9e93-88072e7887a4 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:11.030607579 +0000 UTC m=+49.678521726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls") pod "image-registry-685bcdb8f9-75qqc" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4") : secret "image-registry-tls" not found
Sep 10 21:20:04.846135 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:04.846095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:20:04.846539 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:04.846167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:20:04.846539 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:04.846248    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 21:20:04.846539 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:04.846319    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls podName:e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:12.84630146 +0000 UTC m=+51.494215606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2zz2b" (UID: "e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7") : secret "insights-runtime-extractor-tls" not found
Sep 10 21:20:04.846653 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:04.846641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:20:11.094620 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:11.094588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:20:11.095129 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:11.094659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:20:11.098485 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:11.098458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f48024dc-9b26-46bf-8da8-b9ef5383bf45-metrics-tls\") pod \"dns-default-s8nr2\" (UID: \"f48024dc-9b26-46bf-8da8-b9ef5383bf45\") " pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:20:11.098595 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:11.098492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls\") pod \"image-registry-685bcdb8f9-75qqc\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") " pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:20:11.141689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:11.141662    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:20:11.151955 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:11.151929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:20:11.427908 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:11.427793    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-685bcdb8f9-75qqc"]
Sep 10 21:20:11.431238 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:11.431199    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebabd368_7bcb_4047_9e93_88072e7887a4.slice/crio-35677940ed67f9bd3043b3cf8956b3cd19d7d833439788260c813c301a0124ff WatchSource:0}: Error finding container 35677940ed67f9bd3043b3cf8956b3cd19d7d833439788260c813c301a0124ff: Status 404 returned error can't find the container with id 35677940ed67f9bd3043b3cf8956b3cd19d7d833439788260c813c301a0124ff
Sep 10 21:20:11.444082 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:11.443987    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s8nr2"]
Sep 10 21:20:11.448047 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:11.448021    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48024dc_9b26_46bf_8da8_b9ef5383bf45.slice/crio-1efb8cc33f0755495c36e32227d633947065d9a85dac84bb92633352054c0793 WatchSource:0}: Error finding container 1efb8cc33f0755495c36e32227d633947065d9a85dac84bb92633352054c0793: Status 404 returned error can't find the container with id 1efb8cc33f0755495c36e32227d633947065d9a85dac84bb92633352054c0793
Sep 10 21:20:12.063280 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:12.063227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc" event={"ID":"ebabd368-7bcb-4047-9e93-88072e7887a4","Type":"ContainerStarted","Data":"1ce33efa5ca195416df4b92d59fa736bc76d3b0862117f5bc2a8270bb8cd1b86"}
Sep 10 21:20:12.063490 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:12.063394    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:20:12.063490 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:12.063456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc" event={"ID":"ebabd368-7bcb-4047-9e93-88072e7887a4","Type":"ContainerStarted","Data":"35677940ed67f9bd3043b3cf8956b3cd19d7d833439788260c813c301a0124ff"}
Sep 10 21:20:12.064886 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:12.064864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s8nr2" event={"ID":"f48024dc-9b26-46bf-8da8-b9ef5383bf45","Type":"ContainerStarted","Data":"1efb8cc33f0755495c36e32227d633947065d9a85dac84bb92633352054c0793"}
Sep 10 21:20:12.083907 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:12.083847    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc" podStartSLOduration=50.083828736 podStartE2EDuration="50.083828736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:12.082700012 +0000 UTC m=+50.730614180" watchObservedRunningTime="2026-09-10 21:20:12.083828736 +0000 UTC m=+50.731742901"
Sep 10 21:20:12.910231 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:12.910168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:20:12.912594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:12.912573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2zz2b\" (UID: \"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7\") " pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:20:13.042566 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:13.042535    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2zz2b"
Sep 10 21:20:13.310952 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:13.310916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2zz2b"]
Sep 10 21:20:14.076007 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:14.073832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s8nr2" event={"ID":"f48024dc-9b26-46bf-8da8-b9ef5383bf45","Type":"ContainerStarted","Data":"85504b10f6b6390318ed03f4844af4f6ff272c0c149fa2fe2d01d118bf6c4103"}
Sep 10 21:20:14.076007 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:14.075711    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2zz2b" event={"ID":"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7","Type":"ContainerStarted","Data":"85e9715af6933b695a9e6fc7c4cd497854e4a7be2ebf2b617aa44d467472f3fa"}
Sep 10 21:20:14.076007 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:14.075773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2zz2b" event={"ID":"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7","Type":"ContainerStarted","Data":"68e18d3d9f0680a8f98090932c8a82c057815601d0bcd8c54f294148da507ea0"}
Sep 10 21:20:15.079834 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:15.079794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s8nr2" event={"ID":"f48024dc-9b26-46bf-8da8-b9ef5383bf45","Type":"ContainerStarted","Data":"dfe08714b419d215c35cda05e710a14ba95091a32d82f3d8b5c30ef1f566c5ca"}
Sep 10 21:20:15.080338 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:15.079904    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:20:15.081682 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:15.081661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2zz2b" event={"ID":"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7","Type":"ContainerStarted","Data":"5248bd8a7f1be2df1fc2ae84926d4559be86a4ad36ad285d68e51f31a56dc739"}
Sep 10 21:20:15.096891 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:15.096850    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-s8nr2" podStartSLOduration=17.564757969 podStartE2EDuration="20.096838141s" totalImagesPullingTime="2.532080172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:15.095427991 +0000 UTC m=+53.743342152" watchObservedRunningTime="2026-09-10 21:20:15.096838141 +0000 UTC m=+53.744752309"
Sep 10 21:20:16.086919 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:16.086884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2zz2b" event={"ID":"e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7","Type":"ContainerStarted","Data":"8701003e2e9e084a97349866cbaaa9020b6ad6a76101b210e8adbb51c8a4dd92"}
Sep 10 21:20:16.105264 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:16.105217    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2zz2b" podStartSLOduration=16.931989489 podStartE2EDuration="19.1051981s" totalImagesPullingTime="2.173208611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:16.104285455 +0000 UTC m=+54.752199614" watchObservedRunningTime="2026-09-10 21:20:16.1051981 +0000 UTC m=+54.753112289"
Sep 10 21:20:20.031822 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.031790    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-685bcdb8f9-75qqc"]
Sep 10 21:20:20.134382 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.134347    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f9fbfb55-m5pd5"]
Sep 10 21:20:20.137237 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.137220    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.161318 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.161288    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f9fbfb55-m5pd5"]
Sep 10 21:20:20.267984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.267937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-image-registry-private-configuration\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.267984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.267983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jsz5\" (UniqueName: \"kubernetes.io/projected/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-kube-api-access-9jsz5\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.268194 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.268062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-registry-tls\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.268194 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.268116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-installation-pull-secrets\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.268194 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.268175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-registry-certificates\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.268305 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.268204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-ca-trust-extracted\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.268305 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.268221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-trusted-ca\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.268305 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.268236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-bound-sa-token\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.369464 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.369337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-installation-pull-secrets\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.369464 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.369385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-registry-certificates\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.369689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.369527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-ca-trust-extracted\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.369689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.369562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-trusted-ca\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.369689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.369579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-bound-sa-token\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.369689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.369632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-image-registry-private-configuration\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.369689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.369658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsz5\" (UniqueName: \"kubernetes.io/projected/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-kube-api-access-9jsz5\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.369934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.369736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-registry-tls\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.369934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.369912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-ca-trust-extracted\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.370395 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.370369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-registry-certificates\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.370539 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.370435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-trusted-ca\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.372199 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.372177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-registry-tls\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.372283 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.372244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-image-registry-private-configuration\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.372326 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.372290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-installation-pull-secrets\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.380924 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.380897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-bound-sa-token\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.381918 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.381901    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsz5\" (UniqueName: \"kubernetes.io/projected/2f421fe3-e812-40cb-bfce-a734b0c1c5f8-kube-api-access-9jsz5\") pod \"image-registry-f9fbfb55-m5pd5\" (UID: \"2f421fe3-e812-40cb-bfce-a734b0c1c5f8\") " pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.446133 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.446100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:20.718502 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.718476    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f9fbfb55-m5pd5"]
Sep 10 21:20:20.721860 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:20.721839    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f421fe3_e812_40cb_bfce_a734b0c1c5f8.slice/crio-a5fdc1cb24ac718e3fa77c787f0f5be46e108a3773b80b2239ee3b015ae61156 WatchSource:0}: Error finding container a5fdc1cb24ac718e3fa77c787f0f5be46e108a3773b80b2239ee3b015ae61156: Status 404 returned error can't find the container with id a5fdc1cb24ac718e3fa77c787f0f5be46e108a3773b80b2239ee3b015ae61156
Sep 10 21:20:20.999655 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:20.999584    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cq76m"
Sep 10 21:20:21.103592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:21.103550    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5" event={"ID":"2f421fe3-e812-40cb-bfce-a734b0c1c5f8","Type":"ContainerStarted","Data":"c3a614c5599b87f5d2fac1f0fe8e2edaa6e3db7451511a861e80ac13476d9a71"}
Sep 10 21:20:21.104052 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:21.103611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5" event={"ID":"2f421fe3-e812-40cb-bfce-a734b0c1c5f8","Type":"ContainerStarted","Data":"a5fdc1cb24ac718e3fa77c787f0f5be46e108a3773b80b2239ee3b015ae61156"}
Sep 10 21:20:21.104052 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:21.103716    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:21.124528 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:21.124471    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5" podStartSLOduration=1.124455426 podStartE2EDuration="1.124455426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:21.123611212 +0000 UTC m=+59.771525380" watchObservedRunningTime="2026-09-10 21:20:21.124455426 +0000 UTC m=+59.772369594"
Sep 10 21:20:22.736519 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.736481    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv"]
Sep 10 21:20:22.739816 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.739793    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-5djjw"]
Sep 10 21:20:22.739964 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.739911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv"
Sep 10 21:20:22.741705 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.741682    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-5djjw"
Sep 10 21:20:22.742082 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.742063    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 21:20:22.742187 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.742109    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8f72x\""
Sep 10 21:20:22.743813 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.743791    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 21:20:22.743908 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.743854    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hc984\""
Sep 10 21:20:22.743995 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.743979    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 21:20:22.750257 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.750229    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv"]
Sep 10 21:20:22.765028 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.764999    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-5djjw"]
Sep 10 21:20:22.787687 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.787656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2gvm\" (UniqueName: \"kubernetes.io/projected/261a0890-0436-4126-be0d-16bfa97a14ae-kube-api-access-d2gvm\") pod \"downloads-76f99584f4-5djjw\" (UID: \"261a0890-0436-4126-be0d-16bfa97a14ae\") " pod="openshift-console/downloads-76f99584f4-5djjw"
Sep 10 21:20:22.787865 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.787721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/24e2e7c0-a4eb-462e-b0dc-28d49941db9e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-99zxv\" (UID: \"24e2e7c0-a4eb-462e-b0dc-28d49941db9e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv"
Sep 10 21:20:22.888282 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.888247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/24e2e7c0-a4eb-462e-b0dc-28d49941db9e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-99zxv\" (UID: \"24e2e7c0-a4eb-462e-b0dc-28d49941db9e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv"
Sep 10 21:20:22.888488 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.888334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2gvm\" (UniqueName: \"kubernetes.io/projected/261a0890-0436-4126-be0d-16bfa97a14ae-kube-api-access-d2gvm\") pod \"downloads-76f99584f4-5djjw\" (UID: \"261a0890-0436-4126-be0d-16bfa97a14ae\") " pod="openshift-console/downloads-76f99584f4-5djjw"
Sep 10 21:20:22.888488 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:22.888436    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 21:20:22.888607 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:22.888528    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24e2e7c0-a4eb-462e-b0dc-28d49941db9e-tls-certificates podName:24e2e7c0-a4eb-462e-b0dc-28d49941db9e nodeName:}" failed. No retries permitted until 2026-09-10 21:20:23.388506549 +0000 UTC m=+62.036420714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/24e2e7c0-a4eb-462e-b0dc-28d49941db9e-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-99zxv" (UID: "24e2e7c0-a4eb-462e-b0dc-28d49941db9e") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 21:20:22.900890 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:22.900860    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2gvm\" (UniqueName: \"kubernetes.io/projected/261a0890-0436-4126-be0d-16bfa97a14ae-kube-api-access-d2gvm\") pod \"downloads-76f99584f4-5djjw\" (UID: \"261a0890-0436-4126-be0d-16bfa97a14ae\") " pod="openshift-console/downloads-76f99584f4-5djjw"
Sep 10 21:20:23.056458 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:23.056362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-5djjw"
Sep 10 21:20:23.329362 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:23.329286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-5djjw"]
Sep 10 21:20:23.333317 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:23.333287    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261a0890_0436_4126_be0d_16bfa97a14ae.slice/crio-a1133f3dc74b5984f69693600ec9c0ff5411cf6c8c83408c2f61f21a5fba106f WatchSource:0}: Error finding container a1133f3dc74b5984f69693600ec9c0ff5411cf6c8c83408c2f61f21a5fba106f: Status 404 returned error can't find the container with id a1133f3dc74b5984f69693600ec9c0ff5411cf6c8c83408c2f61f21a5fba106f
Sep 10 21:20:23.391563 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:23.391529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/24e2e7c0-a4eb-462e-b0dc-28d49941db9e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-99zxv\" (UID: \"24e2e7c0-a4eb-462e-b0dc-28d49941db9e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv"
Sep 10 21:20:23.393981 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:23.393964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/24e2e7c0-a4eb-462e-b0dc-28d49941db9e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-99zxv\" (UID: \"24e2e7c0-a4eb-462e-b0dc-28d49941db9e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv"
Sep 10 21:20:23.650617 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:23.650530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv"
Sep 10 21:20:23.918886 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:23.918804    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv"]
Sep 10 21:20:23.922878 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:23.922851    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e2e7c0_a4eb_462e_b0dc_28d49941db9e.slice/crio-9034ba94b2415fd286426232cc801bcfc252b106d746c2a9142894a9906f6cf9 WatchSource:0}: Error finding container 9034ba94b2415fd286426232cc801bcfc252b106d746c2a9142894a9906f6cf9: Status 404 returned error can't find the container with id 9034ba94b2415fd286426232cc801bcfc252b106d746c2a9142894a9906f6cf9
Sep 10 21:20:24.118268 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:24.118225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv" event={"ID":"24e2e7c0-a4eb-462e-b0dc-28d49941db9e","Type":"ContainerStarted","Data":"9034ba94b2415fd286426232cc801bcfc252b106d746c2a9142894a9906f6cf9"}
Sep 10 21:20:24.119390 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:24.119350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-5djjw" event={"ID":"261a0890-0436-4126-be0d-16bfa97a14ae","Type":"ContainerStarted","Data":"a1133f3dc74b5984f69693600ec9c0ff5411cf6c8c83408c2f61f21a5fba106f"}
Sep 10 21:20:25.090934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:25.089491    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s8nr2"
Sep 10 21:20:26.129150 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.129108    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv" event={"ID":"24e2e7c0-a4eb-462e-b0dc-28d49941db9e","Type":"ContainerStarted","Data":"1fc230efee18c4538f0ca81278c9d3c2be9dd08c6ea7b694e35da33a962104ee"}
Sep 10 21:20:26.129631 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.129361    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv"
Sep 10 21:20:26.134909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.134885    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv"
Sep 10 21:20:26.147275 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.147220    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-99zxv" podStartSLOduration=2.906545252 podStartE2EDuration="4.147173103s" totalImagesPullingTime="1.240627851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:26.146572992 +0000 UTC m=+64.794487161" watchObservedRunningTime="2026-09-10 21:20:26.147173103 +0000 UTC m=+64.795087273"
Sep 10 21:20:26.694579 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.694546    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"]
Sep 10 21:20:26.697539 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.697519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:26.699959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.699933    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 21:20:26.700098 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.699995    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 21:20:26.700098 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.700026    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 21:20:26.700098 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.700058    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 21:20:26.700765 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.700749    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7j27d\""
Sep 10 21:20:26.700807 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.700789    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 21:20:26.707936 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.707868    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"]
Sep 10 21:20:26.720463 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.720437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:26.720565 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.720508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:26.720630 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.720612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w4x7\" (UniqueName: \"kubernetes.io/projected/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-kube-api-access-5w4x7\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:26.720673 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.720656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:26.821814 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.821769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w4x7\" (UniqueName: \"kubernetes.io/projected/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-kube-api-access-5w4x7\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:26.822002 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.821840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:26.822002 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.821918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:26.822002 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.821945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:26.822141 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:26.822005    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 10 21:20:26.822141 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:26.822091    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-prometheus-operator-tls podName:86a2798c-6b0d-4f0e-afa2-0cf26a4288c8 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:27.322069214 +0000 UTC m=+65.969983378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-xz2kd" (UID: "86a2798c-6b0d-4f0e-afa2-0cf26a4288c8") : secret "prometheus-operator-tls" not found
Sep 10 21:20:26.822885 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.822832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:26.824737 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.824712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:26.833071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:26.833042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w4x7\" (UniqueName: \"kubernetes.io/projected/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-kube-api-access-5w4x7\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:27.325892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.325851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:27.328895 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.328865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/86a2798c-6b0d-4f0e-afa2-0cf26a4288c8-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xz2kd\" (UID: \"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:27.526695 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.526653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:20:27.528766 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.528739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 21:20:27.540232 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.540208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51decde2-f81d-4c6f-b857-8fd6150f288a-metrics-certs\") pod \"network-metrics-daemon-kgjnr\" (UID: \"51decde2-f81d-4c6f-b857-8fd6150f288a\") " pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:20:27.606378 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.606296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"
Sep 10 21:20:27.639729 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.638744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbh2\" (UniqueName: \"kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2\") pod \"network-check-target-dz7bd\" (UID: \"9dca27a7-8d1a-4c60-bacf-bfad74056541\") " pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:20:27.639729 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.638877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:20:27.641302 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.641272    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 21:20:27.641764 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.641737    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c8fbc\""
Sep 10 21:20:27.642073 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.641944    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 21:20:27.649840 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.649811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kgjnr"
Sep 10 21:20:27.651900 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.651871    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 21:20:27.652521 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.652496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e804440-819b-402d-ba84-f9f0c3ad4aeb-original-pull-secret\") pod \"global-pull-secret-syncer-2r6cp\" (UID: \"7e804440-819b-402d-ba84-f9f0c3ad4aeb\") " pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:20:27.664110 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.664083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbh2\" (UniqueName: \"kubernetes.io/projected/9dca27a7-8d1a-4c60-bacf-bfad74056541-kube-api-access-rxbh2\") pod \"network-check-target-dz7bd\" (UID: \"9dca27a7-8d1a-4c60-bacf-bfad74056541\") " pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:20:27.726036 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.726010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kc896\""
Sep 10 21:20:27.735151 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.735132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:20:27.909231 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.909105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd"]
Sep 10 21:20:27.913212 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:27.913181    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a2798c_6b0d_4f0e_afa2_0cf26a4288c8.slice/crio-5ebd2ce98f1314c261a7ad7cc0d0c74779c0515b84747fa769aa3d83bb383afc WatchSource:0}: Error finding container 5ebd2ce98f1314c261a7ad7cc0d0c74779c0515b84747fa769aa3d83bb383afc: Status 404 returned error can't find the container with id 5ebd2ce98f1314c261a7ad7cc0d0c74779c0515b84747fa769aa3d83bb383afc
Sep 10 21:20:27.932940 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.932913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2r6cp"
Sep 10 21:20:27.938946 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:27.938918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kgjnr"]
Sep 10 21:20:27.943281 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:27.943242    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51decde2_f81d_4c6f_b857_8fd6150f288a.slice/crio-6e51cfeec1a079f268d2fa7caeb527618ba47b1acf3c7e55426a557fb95b6543 WatchSource:0}: Error finding container 6e51cfeec1a079f268d2fa7caeb527618ba47b1acf3c7e55426a557fb95b6543: Status 404 returned error can't find the container with id 6e51cfeec1a079f268d2fa7caeb527618ba47b1acf3c7e55426a557fb95b6543
Sep 10 21:20:28.010766 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:28.010737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dz7bd"]
Sep 10 21:20:28.014025 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:28.013991    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dca27a7_8d1a_4c60_bacf_bfad74056541.slice/crio-cf2a014d86c9104d78cfa48654471e9fcf6fae1612d645016ddabeaa52b7d425 WatchSource:0}: Error finding container cf2a014d86c9104d78cfa48654471e9fcf6fae1612d645016ddabeaa52b7d425: Status 404 returned error can't find the container with id cf2a014d86c9104d78cfa48654471e9fcf6fae1612d645016ddabeaa52b7d425
Sep 10 21:20:28.137788 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:28.137749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dz7bd" event={"ID":"9dca27a7-8d1a-4c60-bacf-bfad74056541","Type":"ContainerStarted","Data":"cf2a014d86c9104d78cfa48654471e9fcf6fae1612d645016ddabeaa52b7d425"}
Sep 10 21:20:28.139182 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:28.139148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd" event={"ID":"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8","Type":"ContainerStarted","Data":"5ebd2ce98f1314c261a7ad7cc0d0c74779c0515b84747fa769aa3d83bb383afc"}
Sep 10 21:20:28.140616 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:28.140581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kgjnr" event={"ID":"51decde2-f81d-4c6f-b857-8fd6150f288a","Type":"ContainerStarted","Data":"6e51cfeec1a079f268d2fa7caeb527618ba47b1acf3c7e55426a557fb95b6543"}
Sep 10 21:20:28.211741 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:28.211697    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2r6cp"]
Sep 10 21:20:28.214958 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:28.214931    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e804440_819b_402d_ba84_f9f0c3ad4aeb.slice/crio-bdfec7a5cc19d542b87978a3f7af86f222c8e58c5db56bba44a91297a20dd9e2 WatchSource:0}: Error finding container bdfec7a5cc19d542b87978a3f7af86f222c8e58c5db56bba44a91297a20dd9e2: Status 404 returned error can't find the container with id bdfec7a5cc19d542b87978a3f7af86f222c8e58c5db56bba44a91297a20dd9e2
Sep 10 21:20:29.148180 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:29.148079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2r6cp" event={"ID":"7e804440-819b-402d-ba84-f9f0c3ad4aeb","Type":"ContainerStarted","Data":"bdfec7a5cc19d542b87978a3f7af86f222c8e58c5db56bba44a91297a20dd9e2"}
Sep 10 21:20:30.039968 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:30.039651    2580 patch_prober.go:28] interesting pod/image-registry-685bcdb8f9-75qqc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 10 21:20:30.039968 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:30.039715    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc" podUID="ebabd368-7bcb-4047-9e93-88072e7887a4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:20:30.159474 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:30.159389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kgjnr" event={"ID":"51decde2-f81d-4c6f-b857-8fd6150f288a","Type":"ContainerStarted","Data":"89e6e7208e49e9ea9dc1f36dac6a0a9fcd72c7e4ed3dbea1a0ec05971a0478fb"}
Sep 10 21:20:30.162593 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:30.162566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd" event={"ID":"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8","Type":"ContainerStarted","Data":"d5650f9d8826f1cd338faf1c0553a7cf50f37af1ff47f56d32823febb84297e4"}
Sep 10 21:20:31.171548 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:31.171374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd" event={"ID":"86a2798c-6b0d-4f0e-afa2-0cf26a4288c8","Type":"ContainerStarted","Data":"08d079d6f43bdff1078a16fc662dbecbf57a240b80c00995ea36b2e8fa88933a"}
Sep 10 21:20:31.173780 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:31.173750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kgjnr" event={"ID":"51decde2-f81d-4c6f-b857-8fd6150f288a","Type":"ContainerStarted","Data":"26c7f824d4717686db70754928476a4f261dc2048f9d0b83a70d59c67a90df8b"}
Sep 10 21:20:31.189536 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:31.189485    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xz2kd" podStartSLOduration=3.108386607 podStartE2EDuration="5.189466879s" totalImagesPullingTime="2.081080272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:31.188713176 +0000 UTC m=+69.836627343" watchObservedRunningTime="2026-09-10 21:20:31.189466879 +0000 UTC m=+69.837381048"
Sep 10 21:20:31.206753 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:31.206698    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kgjnr" podStartSLOduration=68.160842348 podStartE2EDuration="1m10.206677515s" totalImagesPullingTime="2.045835167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:31.205736472 +0000 UTC m=+69.853650640" watchObservedRunningTime="2026-09-10 21:20:31.206677515 +0000 UTC m=+69.854591685"
Sep 10 21:20:32.734574 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.734538    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bc76d5d5b-cdfrd"]
Sep 10 21:20:32.750111 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.750079    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bc76d5d5b-cdfrd"]
Sep 10 21:20:32.750263 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.750179    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.753058 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.753027    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 21:20:32.753585 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.753560    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 21:20:32.754155 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.754127    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9wqqs\""
Sep 10 21:20:32.754365 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.754344    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 21:20:32.754542 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.754522    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 21:20:32.754670 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.754649    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 21:20:32.775745 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.775704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-service-ca\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.775928 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.775747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-oauth-serving-cert\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.775928 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.775782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-oauth-config\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.775928 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.775816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9slw\" (UniqueName: \"kubernetes.io/projected/315ce57e-6d5a-41a7-860e-f629c4f3e5db-kube-api-access-t9slw\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.775928 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.775844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-serving-cert\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.775928 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.775916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-config\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.876285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.876245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-config\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.876285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.876305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-service-ca\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.876565 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.876334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-oauth-serving-cert\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.876565 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.876367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-oauth-config\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.876565 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.876402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9slw\" (UniqueName: \"kubernetes.io/projected/315ce57e-6d5a-41a7-860e-f629c4f3e5db-kube-api-access-t9slw\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.876565 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.876451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-serving-cert\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.877250 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.877222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-service-ca\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.877367 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.877271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-oauth-serving-cert\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.877367 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.877337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-config\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.880270 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.880238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-oauth-config\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.880482 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.880463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-serving-cert\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:32.885958 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:32.885931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9slw\" (UniqueName: \"kubernetes.io/projected/315ce57e-6d5a-41a7-860e-f629c4f3e5db-kube-api-access-t9slw\") pod \"console-7bc76d5d5b-cdfrd\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") " pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:33.062768 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.062666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:33.132238 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.132199    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"]
Sep 10 21:20:33.149677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.149645    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hljtx"]
Sep 10 21:20:33.149851 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.149766    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.159384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.159353    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 21:20:33.159547 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.159518    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 21:20:33.159547 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.159528    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rbpb9\""
Sep 10 21:20:33.161404 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.160533    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 21:20:33.165196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.164595    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"]
Sep 10 21:20:33.165196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.164734    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.168375 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.168357    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 21:20:33.168515 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.168404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 21:20:33.168761 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.168745    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zzvmr\""
Sep 10 21:20:33.177036 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.177010    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 21:20:33.178212 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-wtmp\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.178212 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-metrics-client-ca\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.178357 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkhph\" (UniqueName: \"kubernetes.io/projected/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-kube-api-access-jkhph\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.178357 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-tls\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.178446 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.178446 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-textfile\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.178529 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.178529 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-accelerators-collector-config\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.178529 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.178660 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-sys\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.178660 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.178660 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.178816 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.178816 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpgw\" (UniqueName: \"kubernetes.io/projected/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-api-access-rzpgw\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.178816 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.178774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-root\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.279582 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpgw\" (UniqueName: \"kubernetes.io/projected/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-api-access-rzpgw\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.279582 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-root\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.279838 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-wtmp\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.279838 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-metrics-client-ca\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.279838 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkhph\" (UniqueName: \"kubernetes.io/projected/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-kube-api-access-jkhph\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.279838 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-tls\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.279838 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.279838 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-textfile\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.279838 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.279838 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-accelerators-collector-config\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.279838 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.279838 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-sys\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.280152 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.280152 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.280152 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:33.279940    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 21:20:33.280152 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.279962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.280152 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:20:33.279999    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-tls podName:b6d3dc88-240a-4d69-8fcd-f244e3671ec6 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:33.779984626 +0000 UTC m=+72.427898773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-tls") pod "node-exporter-hljtx" (UID: "b6d3dc88-240a-4d69-8fcd-f244e3671ec6") : secret "node-exporter-tls" not found
Sep 10 21:20:33.280152 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.280101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-wtmp\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.280654 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.280626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-metrics-client-ca\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.280767 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.280748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-accelerators-collector-config\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.280853 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.280834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-textfile\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.281269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.281184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-sys\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.281269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.281233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-root\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.281554 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.281536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.281724 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.281692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.281800 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.281743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.283647 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.283321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.283647 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.283611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.283889 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.283769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.293057 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.293036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpgw\" (UniqueName: \"kubernetes.io/projected/2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a-kube-api-access-rzpgw\") pod \"kube-state-metrics-59fbdd77d-rll9x\" (UID: \"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.295552 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.295530    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkhph\" (UniqueName: \"kubernetes.io/projected/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-kube-api-access-jkhph\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.462574 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.462533    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"
Sep 10 21:20:33.785112 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.785026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-tls\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:33.788067 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:33.788007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6d3dc88-240a-4d69-8fcd-f244e3671ec6-node-exporter-tls\") pod \"node-exporter-hljtx\" (UID: \"b6d3dc88-240a-4d69-8fcd-f244e3671ec6\") " pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:34.075757 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.075726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hljtx"
Sep 10 21:20:34.088083 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:34.088000    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d3dc88_240a_4d69_8fcd_f244e3671ec6.slice/crio-35f319076fea094772f99b9c97f18119b78232e4a4e7ef8978b9b9ad86117f58 WatchSource:0}: Error finding container 35f319076fea094772f99b9c97f18119b78232e4a4e7ef8978b9b9ad86117f58: Status 404 returned error can't find the container with id 35f319076fea094772f99b9c97f18119b78232e4a4e7ef8978b9b9ad86117f58
Sep 10 21:20:34.188543 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.188507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x"]
Sep 10 21:20:34.191688 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.191650    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bc76d5d5b-cdfrd"]
Sep 10 21:20:34.193265 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.193232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2r6cp" event={"ID":"7e804440-819b-402d-ba84-f9f0c3ad4aeb","Type":"ContainerStarted","Data":"e72ca2ad391041e3c3f1f03fc6e90e832b636d7929cd55bb7463b64eb0789971"}
Sep 10 21:20:34.194403 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:34.194361    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd8cc3d_bf22_4f4f_b40a_e9696a5d169a.slice/crio-a471bbe37f1542e680771bd94ce01c24715212e6eb0518e4a5f51d01e422d03c WatchSource:0}: Error finding container a471bbe37f1542e680771bd94ce01c24715212e6eb0518e4a5f51d01e422d03c: Status 404 returned error can't find the container with id a471bbe37f1542e680771bd94ce01c24715212e6eb0518e4a5f51d01e422d03c
Sep 10 21:20:34.196782 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.195683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hljtx" event={"ID":"b6d3dc88-240a-4d69-8fcd-f244e3671ec6","Type":"ContainerStarted","Data":"35f319076fea094772f99b9c97f18119b78232e4a4e7ef8978b9b9ad86117f58"}
Sep 10 21:20:34.198286 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.197846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dz7bd" event={"ID":"9dca27a7-8d1a-4c60-bacf-bfad74056541","Type":"ContainerStarted","Data":"8908518b6996f4c16f1e8d00202681d4a63f0d76424a2c5804e8f6a84dee218c"}
Sep 10 21:20:34.198286 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.198086    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:20:34.200716 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:34.200695    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315ce57e_6d5a_41a7_860e_f629c4f3e5db.slice/crio-8691f794b426d1105355a533d360691d545a7d5b86d1eb42c12bf1b66200d46c WatchSource:0}: Error finding container 8691f794b426d1105355a533d360691d545a7d5b86d1eb42c12bf1b66200d46c: Status 404 returned error can't find the container with id 8691f794b426d1105355a533d360691d545a7d5b86d1eb42c12bf1b66200d46c
Sep 10 21:20:34.240272 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.240202    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2r6cp" podStartSLOduration=66.543667968 podStartE2EDuration="1m12.240180573s" totalImagesPullingTime="5.696512605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:34.215035825 +0000 UTC m=+72.862949994" watchObservedRunningTime="2026-09-10 21:20:34.240180573 +0000 UTC m=+72.888094741"
Sep 10 21:20:34.240923 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.240886    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dz7bd" podStartSLOduration=67.359307283 podStartE2EDuration="1m13.240872319s" totalImagesPullingTime="5.881565036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:34.239988338 +0000 UTC m=+72.887902504" watchObservedRunningTime="2026-09-10 21:20:34.240872319 +0000 UTC m=+72.888786488"
Sep 10 21:20:34.948664 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.948627    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f86df49d6-jhgpq"]
Sep 10 21:20:34.951685 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.951658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:34.958900 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.958837    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 21:20:34.961229 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:34.961207    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f86df49d6-jhgpq"]
Sep 10 21:20:35.002494 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.002455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-trusted-ca-bundle\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.002494 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.002492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-oauth-serving-cert\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.002703 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.002520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnbzv\" (UniqueName: \"kubernetes.io/projected/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-kube-api-access-vnbzv\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.002703 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.002605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-config\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.002703 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.002634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-serving-cert\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.002703 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.002648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-service-ca\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.002703 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.002683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-oauth-config\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.103807 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.103774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-trusted-ca-bundle\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.103807 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.103808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-oauth-serving-cert\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.104037 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.103828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnbzv\" (UniqueName: \"kubernetes.io/projected/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-kube-api-access-vnbzv\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.104037 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.103922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-config\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.104037 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.103981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-serving-cert\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.104037 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.104008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-service-ca\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.104241 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.104049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-oauth-config\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.104547 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.104521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-oauth-serving-cert\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.105007 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.104962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-config\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.105103 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.105075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-service-ca\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.105170 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.105102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-trusted-ca-bundle\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.106931 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.106909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-oauth-config\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.107125 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.107102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-serving-cert\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.111521 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.111502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnbzv\" (UniqueName: \"kubernetes.io/projected/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-kube-api-access-vnbzv\") pod \"console-7f86df49d6-jhgpq\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") " pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.165795 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.165752    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"]
Sep 10 21:20:35.169169 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.169132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.171279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.171258    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 21:20:35.171700 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.171674    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-294tgpcc8tell\""
Sep 10 21:20:35.171811 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.171698    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 21:20:35.171811 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.171775    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 21:20:35.171811 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.171677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 21:20:35.171938 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.171882    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 21:20:35.172065 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.171994    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tfvdl\""
Sep 10 21:20:35.181427 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.181386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"]
Sep 10 21:20:35.203372 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.203269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x" event={"ID":"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a","Type":"ContainerStarted","Data":"a471bbe37f1542e680771bd94ce01c24715212e6eb0518e4a5f51d01e422d03c"}
Sep 10 21:20:35.204395 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.204366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.204533 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.204405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-grpc-tls\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.204533 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.204449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.204533 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.204480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwksg\" (UniqueName: \"kubernetes.io/projected/2669f605-8f71-4e8e-97cd-de5a52a1405e-kube-api-access-wwksg\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.204533 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.204523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-tls\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.204712 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.204620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.204712 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.204672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.204777 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.204727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2669f605-8f71-4e8e-97cd-de5a52a1405e-metrics-client-ca\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.204824 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.204794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bc76d5d5b-cdfrd" event={"ID":"315ce57e-6d5a-41a7-860e-f629c4f3e5db","Type":"ContainerStarted","Data":"8691f794b426d1105355a533d360691d545a7d5b86d1eb42c12bf1b66200d46c"}
Sep 10 21:20:35.263324 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.263283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:35.305519 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.305478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-tls\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.305712 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.305599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.305712 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.305640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.305712 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.305680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2669f605-8f71-4e8e-97cd-de5a52a1405e-metrics-client-ca\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.305712 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.305707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.305909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.305728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-grpc-tls\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.305909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.305752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.305909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.305777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwksg\" (UniqueName: \"kubernetes.io/projected/2669f605-8f71-4e8e-97cd-de5a52a1405e-kube-api-access-wwksg\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.307208 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.307147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2669f605-8f71-4e8e-97cd-de5a52a1405e-metrics-client-ca\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.309144 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.309094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-tls\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.309265 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.309217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-grpc-tls\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.309348 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.309325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.309724 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.309697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.310080 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.310029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.311989 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.311965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2669f605-8f71-4e8e-97cd-de5a52a1405e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.316307 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.316282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwksg\" (UniqueName: \"kubernetes.io/projected/2669f605-8f71-4e8e-97cd-de5a52a1405e-kube-api-access-wwksg\") pod \"thanos-querier-54c9ff55c4-n7n74\" (UID: \"2669f605-8f71-4e8e-97cd-de5a52a1405e\") " pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:35.480090 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:35.479995    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:37.412233 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.412196    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-79b8c94f99-j6jkn"]
Sep 10 21:20:37.414669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.414640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.416791 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.416765    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 21:20:37.416904 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.416813    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 21:20:37.417798 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.417657    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 21:20:37.417798 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.417677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dtm9j\""
Sep 10 21:20:37.418014 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.417992    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-738p1gflpq6tj\""
Sep 10 21:20:37.418117 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.418039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 21:20:37.426282 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.426260    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79b8c94f99-j6jkn"]
Sep 10 21:20:37.522015 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.521970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-secret-metrics-server-client-certs\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.522205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.522023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-secret-metrics-server-tls\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.522205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.522074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-audit-log\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.522205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.522098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-metrics-server-audit-profiles\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.522321 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.522200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.522321 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.522240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-client-ca-bundle\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.522321 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.522294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjbnk\" (UniqueName: \"kubernetes.io/projected/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-kube-api-access-tjbnk\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.623363 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.623326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjbnk\" (UniqueName: \"kubernetes.io/projected/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-kube-api-access-tjbnk\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.623551 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.623397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-secret-metrics-server-client-certs\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.623551 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.623461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-secret-metrics-server-tls\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.623551 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.623505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-audit-log\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.623551 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.623533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-metrics-server-audit-profiles\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.623759 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.623572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.623759 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.623596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-client-ca-bundle\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.624056 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.624004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-audit-log\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.624510 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.624459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.624843 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.624806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-metrics-server-audit-profiles\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.626511 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.626487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-client-ca-bundle\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.626730 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.626705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-secret-metrics-server-tls\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.626824 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.626736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-secret-metrics-server-client-certs\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.632310 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.632281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjbnk\" (UniqueName: \"kubernetes.io/projected/d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8-kube-api-access-tjbnk\") pod \"metrics-server-79b8c94f99-j6jkn\" (UID: \"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8\") " pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.726649 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.726608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:37.867211 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:37.867176    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bc76d5d5b-cdfrd"]
Sep 10 21:20:39.286556 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.285652    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:20:39.289406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.289383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.292854 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.292823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 21:20:39.293078 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.293063    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 21:20:39.293996 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.293838    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 21:20:39.293996 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.293901    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 21:20:39.294227 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.294075    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 21:20:39.295831 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.293837    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 21:20:39.295831 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.295255    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 21:20:39.296066 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.296043    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gc7bc\""
Sep 10 21:20:39.296205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.296111    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3td07tp6mdq28\""
Sep 10 21:20:39.296878 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.296368    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 21:20:39.296878 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.296592    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 21:20:39.296878 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.296669    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 21:20:39.296878 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.296724    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 21:20:39.296878 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.296799    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 21:20:39.302047 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.302023    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 21:20:39.304066 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.303840    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 21:20:39.326998 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.326893    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:20:39.341317 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.341491 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.341491 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.341491 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.341715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.341715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.341715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.341715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.341715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gxx\" (UniqueName: \"kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-kube-api-access-x2gxx\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.341715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.341715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.342053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.342053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.342053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.342053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.342053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.342053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.342053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.342053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.342053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.341989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.442628 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.442823 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.442823 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.442823 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.442823 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.442823 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.442823 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.442823 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443165 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443165 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443165 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443165 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gxx\" (UniqueName: \"kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-kube-api-access-x2gxx\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443165 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443165 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.442994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443165 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.443022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443165 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.443053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443165 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.443085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443165 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.443132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443580 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.443180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443580 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.443203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443580 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.443278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.443580 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.443278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.446381 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.446357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.446976 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.446708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.446976 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.446790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.446976 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.446900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.447145 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.447099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.447185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.447165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.447399 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.447371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.448131 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.448105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.448223 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.448200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.448767 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.448745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.448932 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.448906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.449078 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.449046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.451726 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.451399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.451726 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.451567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.451726 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.451670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.452838 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.452819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.452934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.452900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.457950 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.457927    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gxx\" (UniqueName: \"kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-kube-api-access-x2gxx\") pod \"prometheus-k8s-0\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:39.605397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:39.605308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:40.038094 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:40.038056    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:20:41.706699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:41.706599    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f86df49d6-jhgpq"]
Sep 10 21:20:41.717778 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:41.717741    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c61a6a_658f_4bd4_aba1_b78bdb0e2513.slice/crio-b1f78a55bd5138024c574990e7abf39920799f81dd949d05525aa27d29da7e82 WatchSource:0}: Error finding container b1f78a55bd5138024c574990e7abf39920799f81dd949d05525aa27d29da7e82: Status 404 returned error can't find the container with id b1f78a55bd5138024c574990e7abf39920799f81dd949d05525aa27d29da7e82
Sep 10 21:20:41.725915 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:41.725837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"]
Sep 10 21:20:41.727826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:41.727747    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79b8c94f99-j6jkn"]
Sep 10 21:20:41.729044 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:41.729009    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2669f605_8f71_4e8e_97cd_de5a52a1405e.slice/crio-9b893c6aece72f12314c49a036082f1eb3edbf7a7bb9ec605492ff5622d47803 WatchSource:0}: Error finding container 9b893c6aece72f12314c49a036082f1eb3edbf7a7bb9ec605492ff5622d47803: Status 404 returned error can't find the container with id 9b893c6aece72f12314c49a036082f1eb3edbf7a7bb9ec605492ff5622d47803
Sep 10 21:20:41.732878 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:41.732825    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27eb360_a4ed_47cc_9bb3_84d8c3dbaeb8.slice/crio-ae2930779d9b5b693e4ad3a6d4cd7e57b424bcb10fc181b82a63be1cde36311b WatchSource:0}: Error finding container ae2930779d9b5b693e4ad3a6d4cd7e57b424bcb10fc181b82a63be1cde36311b: Status 404 returned error can't find the container with id ae2930779d9b5b693e4ad3a6d4cd7e57b424bcb10fc181b82a63be1cde36311b
Sep 10 21:20:41.734473 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:41.734452    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:20:41.742139 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:20:41.742106    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0eeb5f_c850_47f9_97e9_1528f3c3478f.slice/crio-310cba4433885c8afdc4dff3e00335480ea5b385841ac48b6236eefe2075e71a WatchSource:0}: Error finding container 310cba4433885c8afdc4dff3e00335480ea5b385841ac48b6236eefe2075e71a: Status 404 returned error can't find the container with id 310cba4433885c8afdc4dff3e00335480ea5b385841ac48b6236eefe2075e71a
Sep 10 21:20:42.113671 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:42.113562    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f9fbfb55-m5pd5"
Sep 10 21:20:42.241259 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:42.238394    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-5djjw" event={"ID":"261a0890-0436-4126-be0d-16bfa97a14ae","Type":"ContainerStarted","Data":"e7c9dc3c62c1cd0c2231e67b1a417022686e139a38fa5c85f470d08ef31b66bd"}
Sep 10 21:20:42.241259 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:42.239512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-5djjw"
Sep 10 21:20:42.242356 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:42.242324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f86df49d6-jhgpq" event={"ID":"53c61a6a-658f-4bd4-aba1-b78bdb0e2513","Type":"ContainerStarted","Data":"b1f78a55bd5138024c574990e7abf39920799f81dd949d05525aa27d29da7e82"}
Sep 10 21:20:42.245354 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:42.245328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerStarted","Data":"310cba4433885c8afdc4dff3e00335480ea5b385841ac48b6236eefe2075e71a"}
Sep 10 21:20:42.247908 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:42.247883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74" event={"ID":"2669f605-8f71-4e8e-97cd-de5a52a1405e","Type":"ContainerStarted","Data":"9b893c6aece72f12314c49a036082f1eb3edbf7a7bb9ec605492ff5622d47803"}
Sep 10 21:20:42.249682 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:42.249661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn" event={"ID":"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8","Type":"ContainerStarted","Data":"ae2930779d9b5b693e4ad3a6d4cd7e57b424bcb10fc181b82a63be1cde36311b"}
Sep 10 21:20:42.250140 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:42.250073    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-5djjw"
Sep 10 21:20:42.258195 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:42.258153    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-5djjw" podStartSLOduration=2.082765809 podStartE2EDuration="20.258140021s" totalImagesPullingTime="18.175374212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:42.258099293 +0000 UTC m=+80.906013484" watchObservedRunningTime="2026-09-10 21:20:42.258140021 +0000 UTC m=+80.906054192"
Sep 10 21:20:45.052442 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:45.052154    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc" podUID="ebabd368-7bcb-4047-9e93-88072e7887a4" containerName="registry" containerID="cri-o://1ce33efa5ca195416df4b92d59fa736bc76d3b0862117f5bc2a8270bb8cd1b86" gracePeriod=30
Sep 10 21:20:47.278224 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:47.278191    2580 generic.go:356] "Generic (PLEG): container finished" podID="ebabd368-7bcb-4047-9e93-88072e7887a4" containerID="1ce33efa5ca195416df4b92d59fa736bc76d3b0862117f5bc2a8270bb8cd1b86" exitCode=0
Sep 10 21:20:47.278584 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:47.278269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc" event={"ID":"ebabd368-7bcb-4047-9e93-88072e7887a4","Type":"ContainerDied","Data":"1ce33efa5ca195416df4b92d59fa736bc76d3b0862117f5bc2a8270bb8cd1b86"}
Sep 10 21:20:48.619969 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.619938    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:20:48.750088 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.750062    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbmhh\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-kube-api-access-fbmhh\") pod \"ebabd368-7bcb-4047-9e93-88072e7887a4\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") "
Sep 10 21:20:48.750189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.750109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-bound-sa-token\") pod \"ebabd368-7bcb-4047-9e93-88072e7887a4\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") "
Sep 10 21:20:48.750189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.750154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls\") pod \"ebabd368-7bcb-4047-9e93-88072e7887a4\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") "
Sep 10 21:20:48.750189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.750180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-image-registry-private-configuration\") pod \"ebabd368-7bcb-4047-9e93-88072e7887a4\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") "
Sep 10 21:20:48.750348 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.750212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-certificates\") pod \"ebabd368-7bcb-4047-9e93-88072e7887a4\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") "
Sep 10 21:20:48.750348 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.750244    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebabd368-7bcb-4047-9e93-88072e7887a4-ca-trust-extracted\") pod \"ebabd368-7bcb-4047-9e93-88072e7887a4\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") "
Sep 10 21:20:48.750348 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.750289    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-installation-pull-secrets\") pod \"ebabd368-7bcb-4047-9e93-88072e7887a4\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") "
Sep 10 21:20:48.750348 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.750323    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-trusted-ca\") pod \"ebabd368-7bcb-4047-9e93-88072e7887a4\" (UID: \"ebabd368-7bcb-4047-9e93-88072e7887a4\") "
Sep 10 21:20:48.752436 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.750926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ebabd368-7bcb-4047-9e93-88072e7887a4" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:20:48.752436 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.751044    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ebabd368-7bcb-4047-9e93-88072e7887a4" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:20:48.754081 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.754049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-kube-api-access-fbmhh" (OuterVolumeSpecName: "kube-api-access-fbmhh") pod "ebabd368-7bcb-4047-9e93-88072e7887a4" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4"). InnerVolumeSpecName "kube-api-access-fbmhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:48.762017 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.761945    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ebabd368-7bcb-4047-9e93-88072e7887a4" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:20:48.762017 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.761942    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ebabd368-7bcb-4047-9e93-88072e7887a4" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:48.762017 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.762000    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ebabd368-7bcb-4047-9e93-88072e7887a4" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:48.763805 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.763776    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebabd368-7bcb-4047-9e93-88072e7887a4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ebabd368-7bcb-4047-9e93-88072e7887a4" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:48.763883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.763799    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ebabd368-7bcb-4047-9e93-88072e7887a4" (UID: "ebabd368-7bcb-4047-9e93-88072e7887a4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:20:48.851189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.851048    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbmhh\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-kube-api-access-fbmhh\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:20:48.851189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.851081    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-bound-sa-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:20:48.851189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.851096    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-tls\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:20:48.851189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.851109    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-image-registry-private-configuration\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:20:48.851189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.851125    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-registry-certificates\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:20:48.851189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.851140    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebabd368-7bcb-4047-9e93-88072e7887a4-ca-trust-extracted\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:20:48.851189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.851155    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebabd368-7bcb-4047-9e93-88072e7887a4-installation-pull-secrets\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:20:48.851189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.851168    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebabd368-7bcb-4047-9e93-88072e7887a4-trusted-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:20:48.935618 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:48.935551    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f86df49d6-jhgpq"]
Sep 10 21:20:49.290659 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.289515    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerID="5ee6a605c85b3c189af497b84ba7a7df80c62fbf06e9d375c918125a7d81166c" exitCode=0
Sep 10 21:20:49.290659 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.289632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerDied","Data":"5ee6a605c85b3c189af497b84ba7a7df80c62fbf06e9d375c918125a7d81166c"}
Sep 10 21:20:49.294774 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.294315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74" event={"ID":"2669f605-8f71-4e8e-97cd-de5a52a1405e","Type":"ContainerStarted","Data":"0a1ce3ba9be6a5ca1f61fa567276e315c54040c5c18a4e9cd013156ad515cbc9"}
Sep 10 21:20:49.294774 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.294386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74" event={"ID":"2669f605-8f71-4e8e-97cd-de5a52a1405e","Type":"ContainerStarted","Data":"5f8c6e04e6fe66ec2063d12ca274a16032fcc5f3babbf2a11eab4b0016427ae5"}
Sep 10 21:20:49.294774 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.294446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74" event={"ID":"2669f605-8f71-4e8e-97cd-de5a52a1405e","Type":"ContainerStarted","Data":"1c81e2f1c8cc560f2879f4d8d7309f6445fd8dd3289a3ed895d8f77e7672d5f4"}
Sep 10 21:20:49.298605 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.298276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bc76d5d5b-cdfrd" event={"ID":"315ce57e-6d5a-41a7-860e-f629c4f3e5db","Type":"ContainerStarted","Data":"9d251c5ac786755c632539226e0f2f21fd36b7a3e87999fd0d156de598846e7e"}
Sep 10 21:20:49.301611 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.301557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn" event={"ID":"d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8","Type":"ContainerStarted","Data":"8c5b9351d9bd2b4becc489a60e45a4bf6576569405cd71325c34f306d019d40b"}
Sep 10 21:20:49.305045 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.303939    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc"
Sep 10 21:20:49.305045 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.303952    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685bcdb8f9-75qqc" event={"ID":"ebabd368-7bcb-4047-9e93-88072e7887a4","Type":"ContainerDied","Data":"35677940ed67f9bd3043b3cf8956b3cd19d7d833439788260c813c301a0124ff"}
Sep 10 21:20:49.305045 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.304009    2580 scope.go:117] "RemoveContainer" containerID="1ce33efa5ca195416df4b92d59fa736bc76d3b0862117f5bc2a8270bb8cd1b86"
Sep 10 21:20:49.312380 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.311733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x" event={"ID":"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a","Type":"ContainerStarted","Data":"af93c6efb468e301dd63e24cdcaaf20f775ae753020cd726acbbdccdee4c6202"}
Sep 10 21:20:49.312380 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.311787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x" event={"ID":"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a","Type":"ContainerStarted","Data":"9ed64183b5318125a3be693f1c64e8902ef0c0dee8ab8d260251e5fecf05ff1f"}
Sep 10 21:20:49.312380 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.311815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x" event={"ID":"2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a","Type":"ContainerStarted","Data":"fdf5b1fff158469bbb2a686f43f4e3e96e70f3c6a502aaf165ba2cbfa04a3a89"}
Sep 10 21:20:49.315113 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.314526    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6d3dc88-240a-4d69-8fcd-f244e3671ec6" containerID="07c3d5a0264eb3bb40b536c6037f1642adbe02ed5e9b9e884a946c864ab0a135" exitCode=0
Sep 10 21:20:49.315113 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.314618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hljtx" event={"ID":"b6d3dc88-240a-4d69-8fcd-f244e3671ec6","Type":"ContainerDied","Data":"07c3d5a0264eb3bb40b536c6037f1642adbe02ed5e9b9e884a946c864ab0a135"}
Sep 10 21:20:49.318833 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.318815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f86df49d6-jhgpq" event={"ID":"53c61a6a-658f-4bd4-aba1-b78bdb0e2513","Type":"ContainerStarted","Data":"1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af"}
Sep 10 21:20:49.386342 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.386283    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rll9x" podStartSLOduration=2.005007105 podStartE2EDuration="16.38626168s" totalImagesPullingTime="14.381254575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:49.384615901 +0000 UTC m=+88.032530069" watchObservedRunningTime="2026-09-10 21:20:49.38626168 +0000 UTC m=+88.034175850"
Sep 10 21:20:49.428115 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.428073    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f86df49d6-jhgpq" podStartSLOduration=8.432342209 podStartE2EDuration="15.428052745s" totalImagesPullingTime="6.995710536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:49.408629196 +0000 UTC m=+88.056543365" watchObservedRunningTime="2026-09-10 21:20:49.428052745 +0000 UTC m=+88.075966913"
Sep 10 21:20:49.428761 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.428700    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bc76d5d5b-cdfrd" podStartSLOduration=2.906113327 podStartE2EDuration="17.428689916s" totalImagesPullingTime="14.522576589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:49.42649176 +0000 UTC m=+88.074405928" watchObservedRunningTime="2026-09-10 21:20:49.428689916 +0000 UTC m=+88.076604083"
Sep 10 21:20:49.444851 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.444799    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn" podStartSLOduration=5.463502968 podStartE2EDuration="12.444776023s" totalImagesPullingTime="6.981273055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:49.443689327 +0000 UTC m=+88.091603529" watchObservedRunningTime="2026-09-10 21:20:49.444776023 +0000 UTC m=+88.092690192"
Sep 10 21:20:49.460863 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.460836    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-685bcdb8f9-75qqc"]
Sep 10 21:20:49.466800 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.466773    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-685bcdb8f9-75qqc"]
Sep 10 21:20:49.821654 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:49.821608    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebabd368-7bcb-4047-9e93-88072e7887a4" path="/var/lib/kubelet/pods/ebabd368-7bcb-4047-9e93-88072e7887a4/volumes"
Sep 10 21:20:50.336743 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:50.336611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hljtx" event={"ID":"b6d3dc88-240a-4d69-8fcd-f244e3671ec6","Type":"ContainerStarted","Data":"dee62f7861f5df9ec1ce5e73a2986ed1cb32c5e38364991902e80180bf3252d9"}
Sep 10 21:20:50.337595 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:50.337479    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hljtx" event={"ID":"b6d3dc88-240a-4d69-8fcd-f244e3671ec6","Type":"ContainerStarted","Data":"5915b3b8cad6fea642d6765f4291f1dc1d6f5ada3e3b3c7d1c9343eaadb648c7"}
Sep 10 21:20:50.359785 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:50.359730    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hljtx" podStartSLOduration=3.23281261 podStartE2EDuration="17.359706753s" totalImagesPullingTime="13.906381587s" totalInitContainerRuntime="220.512556ms" isStatefulPod=true podCreationTimestamp="2026-09-10 21:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:50.357202971 +0000 UTC m=+89.005117141" watchObservedRunningTime="2026-09-10 21:20:50.359706753 +0000 UTC m=+89.007620923"
Sep 10 21:20:51.343693 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:51.343657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74" event={"ID":"2669f605-8f71-4e8e-97cd-de5a52a1405e","Type":"ContainerStarted","Data":"53d3ecf54af3d21c7db34be830cb8db6c4d6f10c4bbe18344e7728ec1663aac7"}
Sep 10 21:20:51.344190 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:51.343723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74" event={"ID":"2669f605-8f71-4e8e-97cd-de5a52a1405e","Type":"ContainerStarted","Data":"fe53a5cdde0bb18627b0ed89054ef4ffea6fe3433464086633cce5a9d7ec58a7"}
Sep 10 21:20:51.344190 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:51.343750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74" event={"ID":"2669f605-8f71-4e8e-97cd-de5a52a1405e","Type":"ContainerStarted","Data":"0469baefdd927e34e404912d89da24b760e0572c27a8897b896855a4424dd526"}
Sep 10 21:20:51.344190 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:51.343938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:51.370796 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:51.370742    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74" podStartSLOduration=8.210109555 podStartE2EDuration="16.370723862s" totalImagesPullingTime="8.160614307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:51.368614932 +0000 UTC m=+90.016529101" watchObservedRunningTime="2026-09-10 21:20:51.370723862 +0000 UTC m=+90.018638029"
Sep 10 21:20:53.063181 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:53.063145    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:20:54.362337 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:54.362299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerStarted","Data":"d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766"}
Sep 10 21:20:54.362788 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:54.362364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerStarted","Data":"92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6"}
Sep 10 21:20:54.362788 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:54.362389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerStarted","Data":"e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9"}
Sep 10 21:20:54.362788 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:54.362433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerStarted","Data":"4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352"}
Sep 10 21:20:54.362788 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:54.362465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerStarted","Data":"af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c"}
Sep 10 21:20:54.362788 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:54.362489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerStarted","Data":"c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d"}
Sep 10 21:20:54.397477 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:54.397427    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.336832446 podStartE2EDuration="15.397392746s" totalImagesPullingTime="11.002013957s" totalInitContainerRuntime="58.546343ms" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:54.395834396 +0000 UTC m=+93.043748576" watchObservedRunningTime="2026-09-10 21:20:54.397392746 +0000 UTC m=+93.045306914"
Sep 10 21:20:54.606448 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:54.606407    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:20:55.263669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:55.263634    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:20:57.356493 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:57.356464    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-54c9ff55c4-n7n74"
Sep 10 21:20:57.727561 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:57.727525    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:20:57.727726 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:20:57.727611    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:21:05.207459 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:05.207396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dz7bd"
Sep 10 21:21:14.323654 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.323609    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bc76d5d5b-cdfrd" podUID="315ce57e-6d5a-41a7-860e-f629c4f3e5db" containerName="console" containerID="cri-o://9d251c5ac786755c632539226e0f2f21fd36b7a3e87999fd0d156de598846e7e" gracePeriod=15
Sep 10 21:21:14.441258 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.441230    2580 generic.go:356] "Generic (PLEG): container finished" podID="315ce57e-6d5a-41a7-860e-f629c4f3e5db" containerID="9d251c5ac786755c632539226e0f2f21fd36b7a3e87999fd0d156de598846e7e" exitCode=0
Sep 10 21:21:14.441359 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.441272    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bc76d5d5b-cdfrd" event={"ID":"315ce57e-6d5a-41a7-860e-f629c4f3e5db","Type":"ContainerDied","Data":"9d251c5ac786755c632539226e0f2f21fd36b7a3e87999fd0d156de598846e7e"}
Sep 10 21:21:14.562921 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.562898    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:21:14.583974 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.583915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-oauth-config\") pod \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") "
Sep 10 21:21:14.583974 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.583953    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-serving-cert\") pod \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") "
Sep 10 21:21:14.584145 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.584012    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9slw\" (UniqueName: \"kubernetes.io/projected/315ce57e-6d5a-41a7-860e-f629c4f3e5db-kube-api-access-t9slw\") pod \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") "
Sep 10 21:21:14.584145 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.584044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-config\") pod \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") "
Sep 10 21:21:14.584145 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.584110    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-service-ca\") pod \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") "
Sep 10 21:21:14.584262 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.584147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-oauth-serving-cert\") pod \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\" (UID: \"315ce57e-6d5a-41a7-860e-f629c4f3e5db\") "
Sep 10 21:21:14.584742 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.584715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-service-ca" (OuterVolumeSpecName: "service-ca") pod "315ce57e-6d5a-41a7-860e-f629c4f3e5db" (UID: "315ce57e-6d5a-41a7-860e-f629c4f3e5db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:14.584742 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.584721    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "315ce57e-6d5a-41a7-860e-f629c4f3e5db" (UID: "315ce57e-6d5a-41a7-860e-f629c4f3e5db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:14.584925 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.584839    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-config" (OuterVolumeSpecName: "console-config") pod "315ce57e-6d5a-41a7-860e-f629c4f3e5db" (UID: "315ce57e-6d5a-41a7-860e-f629c4f3e5db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:14.586677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.586648    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "315ce57e-6d5a-41a7-860e-f629c4f3e5db" (UID: "315ce57e-6d5a-41a7-860e-f629c4f3e5db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:14.586811 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.586743    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "315ce57e-6d5a-41a7-860e-f629c4f3e5db" (UID: "315ce57e-6d5a-41a7-860e-f629c4f3e5db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:14.586811 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.586783    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315ce57e-6d5a-41a7-860e-f629c4f3e5db-kube-api-access-t9slw" (OuterVolumeSpecName: "kube-api-access-t9slw") pod "315ce57e-6d5a-41a7-860e-f629c4f3e5db" (UID: "315ce57e-6d5a-41a7-860e-f629c4f3e5db"). InnerVolumeSpecName "kube-api-access-t9slw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:14.685250 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.685208    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:14.685250 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.685244    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-oauth-serving-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:14.685250 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.685255    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-oauth-config\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:14.685250 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.685264    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-serving-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:14.685534 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.685274    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9slw\" (UniqueName: \"kubernetes.io/projected/315ce57e-6d5a-41a7-860e-f629c4f3e5db-kube-api-access-t9slw\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:14.685534 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:14.685283    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/315ce57e-6d5a-41a7-860e-f629c4f3e5db-console-config\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:15.370256 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.370214    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f86df49d6-jhgpq" podUID="53c61a6a-658f-4bd4-aba1-b78bdb0e2513" containerName="console" containerID="cri-o://1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af" gracePeriod=15
Sep 10 21:21:15.446884 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.446850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bc76d5d5b-cdfrd" event={"ID":"315ce57e-6d5a-41a7-860e-f629c4f3e5db","Type":"ContainerDied","Data":"8691f794b426d1105355a533d360691d545a7d5b86d1eb42c12bf1b66200d46c"}
Sep 10 21:21:15.446884 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.446883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bc76d5d5b-cdfrd"
Sep 10 21:21:15.447068 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.446920    2580 scope.go:117] "RemoveContainer" containerID="9d251c5ac786755c632539226e0f2f21fd36b7a3e87999fd0d156de598846e7e"
Sep 10 21:21:15.468327 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.468300    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bc76d5d5b-cdfrd"]
Sep 10 21:21:15.472761 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.472739    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bc76d5d5b-cdfrd"]
Sep 10 21:21:15.612872 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.612847    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:21:15.694448 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.694399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-config\") pod \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") "
Sep 10 21:21:15.694615 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.694482    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-trusted-ca-bundle\") pod \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") "
Sep 10 21:21:15.694615 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.694515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-service-ca\") pod \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") "
Sep 10 21:21:15.694615 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.694545    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnbzv\" (UniqueName: \"kubernetes.io/projected/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-kube-api-access-vnbzv\") pod \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") "
Sep 10 21:21:15.694615 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.694573    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-oauth-serving-cert\") pod \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") "
Sep 10 21:21:15.694826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.694700    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-serving-cert\") pod \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") "
Sep 10 21:21:15.694826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.694741    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-oauth-config\") pod \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\" (UID: \"53c61a6a-658f-4bd4-aba1-b78bdb0e2513\") "
Sep 10 21:21:15.694919 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.694752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-config" (OuterVolumeSpecName: "console-config") pod "53c61a6a-658f-4bd4-aba1-b78bdb0e2513" (UID: "53c61a6a-658f-4bd4-aba1-b78bdb0e2513"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:15.694979 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.694960    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-service-ca" (OuterVolumeSpecName: "service-ca") pod "53c61a6a-658f-4bd4-aba1-b78bdb0e2513" (UID: "53c61a6a-658f-4bd4-aba1-b78bdb0e2513"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:15.694979 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.694967    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "53c61a6a-658f-4bd4-aba1-b78bdb0e2513" (UID: "53c61a6a-658f-4bd4-aba1-b78bdb0e2513"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:15.695084 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.694989    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "53c61a6a-658f-4bd4-aba1-b78bdb0e2513" (UID: "53c61a6a-658f-4bd4-aba1-b78bdb0e2513"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:15.695084 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.695054    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:15.695084 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.695071    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-oauth-serving-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:15.695227 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.695085    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-config\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:15.696783 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.696759    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-kube-api-access-vnbzv" (OuterVolumeSpecName: "kube-api-access-vnbzv") pod "53c61a6a-658f-4bd4-aba1-b78bdb0e2513" (UID: "53c61a6a-658f-4bd4-aba1-b78bdb0e2513"). InnerVolumeSpecName "kube-api-access-vnbzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:15.696923 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.696908    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "53c61a6a-658f-4bd4-aba1-b78bdb0e2513" (UID: "53c61a6a-658f-4bd4-aba1-b78bdb0e2513"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:15.696997 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.696981    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "53c61a6a-658f-4bd4-aba1-b78bdb0e2513" (UID: "53c61a6a-658f-4bd4-aba1-b78bdb0e2513"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:15.795571 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.795538    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnbzv\" (UniqueName: \"kubernetes.io/projected/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-kube-api-access-vnbzv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:15.795571 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.795567    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-serving-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:15.795571 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.795577    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-console-oauth-config\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:15.795788 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.795586    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53c61a6a-658f-4bd4-aba1-b78bdb0e2513-trusted-ca-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:15.819291 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:15.819261    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="315ce57e-6d5a-41a7-860e-f629c4f3e5db" path="/var/lib/kubelet/pods/315ce57e-6d5a-41a7-860e-f629c4f3e5db/volumes"
Sep 10 21:21:16.452935 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:16.452902    2580 generic.go:356] "Generic (PLEG): container finished" podID="53c61a6a-658f-4bd4-aba1-b78bdb0e2513" containerID="1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af" exitCode=0
Sep 10 21:21:16.453383 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:16.452955    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f86df49d6-jhgpq"
Sep 10 21:21:16.453383 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:16.452966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f86df49d6-jhgpq" event={"ID":"53c61a6a-658f-4bd4-aba1-b78bdb0e2513","Type":"ContainerDied","Data":"1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af"}
Sep 10 21:21:16.453383 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:16.453027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f86df49d6-jhgpq" event={"ID":"53c61a6a-658f-4bd4-aba1-b78bdb0e2513","Type":"ContainerDied","Data":"b1f78a55bd5138024c574990e7abf39920799f81dd949d05525aa27d29da7e82"}
Sep 10 21:21:16.453383 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:16.453046    2580 scope.go:117] "RemoveContainer" containerID="1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af"
Sep 10 21:21:16.461181 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:16.461163    2580 scope.go:117] "RemoveContainer" containerID="1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af"
Sep 10 21:21:16.461468 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:21:16.461445    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af\": container with ID starting with 1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af not found: ID does not exist" containerID="1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af"
Sep 10 21:21:16.461542 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:16.461480    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af"} err="failed to get container status \"1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af\": rpc error: code = NotFound desc = could not find container \"1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af\": container with ID starting with 1870a2bdd880ee37b099d17358d55b11891f4f58083163a26d1d9b1772a2f6af not found: ID does not exist"
Sep 10 21:21:16.469504 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:16.469483    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f86df49d6-jhgpq"]
Sep 10 21:21:16.473333 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:16.473314    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f86df49d6-jhgpq"]
Sep 10 21:21:17.732468 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:17.732439    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:21:17.736634 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:17.736610    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-79b8c94f99-j6jkn"
Sep 10 21:21:17.819560 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:17.819529    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c61a6a-658f-4bd4-aba1-b78bdb0e2513" path="/var/lib/kubelet/pods/53c61a6a-658f-4bd4-aba1-b78bdb0e2513/volumes"
Sep 10 21:21:39.605734 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:39.605690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:39.631042 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:39.631015    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:40.564980 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:40.564942    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:57.631553 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:57.631519    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:21:57.632101 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:57.631806    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="prometheus" containerID="cri-o://c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d" gracePeriod=600
Sep 10 21:21:57.632101 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:57.631856    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy" containerID="cri-o://92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6" gracePeriod=600
Sep 10 21:21:57.632101 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:57.631873    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766" gracePeriod=600
Sep 10 21:21:57.632101 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:57.631901    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="thanos-sidecar" containerID="cri-o://4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352" gracePeriod=600
Sep 10 21:21:57.632101 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:57.631931    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy-web" containerID="cri-o://e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9" gracePeriod=600
Sep 10 21:21:57.632101 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:57.631963    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="config-reloader" containerID="cri-o://af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c" gracePeriod=600
Sep 10 21:21:58.621728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.621692    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerID="d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766" exitCode=0
Sep 10 21:21:58.621728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.621718    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerID="92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6" exitCode=0
Sep 10 21:21:58.621728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.621725    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerID="4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352" exitCode=0
Sep 10 21:21:58.621728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.621732    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerID="af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c" exitCode=0
Sep 10 21:21:58.621728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.621741    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerID="c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d" exitCode=0
Sep 10 21:21:58.622061 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.621754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerDied","Data":"d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766"}
Sep 10 21:21:58.622061 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.621818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerDied","Data":"92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6"}
Sep 10 21:21:58.622061 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.621844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerDied","Data":"4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352"}
Sep 10 21:21:58.622061 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.621869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerDied","Data":"af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c"}
Sep 10 21:21:58.622061 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.621893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerDied","Data":"c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d"}
Sep 10 21:21:58.865889 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.865864    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:58.961241 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961211    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-kubelet-serving-ca-bundle\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:58.961431 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961279    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-grpc-tls\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:58.961431 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961350    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config-out\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:58.961431 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961381    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:58.961594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961430    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-thanos-prometheus-http-client-file\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:58.961594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961459    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-tls-assets\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:58.961594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961490    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-metrics-client-ca\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:58.961594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-db\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:58.961594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-kube-rbac-proxy\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:58.961594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961589    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-metrics-client-certs\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:58.961895 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961620    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2gxx\" (UniqueName: \"kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-kube-api-access-x2gxx\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:58.961985 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:58.962043 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.961951    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:58.964720 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.964628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config-out" (OuterVolumeSpecName: "config-out") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:58.964885 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.964729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-kube-api-access-x2gxx" (OuterVolumeSpecName: "kube-api-access-x2gxx") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "kube-api-access-x2gxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:58.964885 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.964738    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:58.964885 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.964849    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:58.964885 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.964874    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:58.965197 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.965161    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:58.965434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.965394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:58.965692 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.965670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:58.965961 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:58.965940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:59.062429 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.062388    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-0\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:59.062592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.062451    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-serving-certs-ca-bundle\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:59.062592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.062479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:59.062592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.062549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-2\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:59.062784 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.062615    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:59.062784 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.062650    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-web-config\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:59.062784 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.062685    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-trusted-ca-bundle\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:59.062784 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.062714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-tls\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:59.062784 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.062738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-1\") pod \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\" (UID: \"fb0eeb5f-c850-47f9-97e9-1528f3c3478f\") "
Sep 10 21:21:59.063009 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.062826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:59.063009 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.062844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:59.063148 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063122    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:59.063204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063130    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config-out\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063130    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:59.063204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063169    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063187    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063202    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-tls-assets\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063218    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-metrics-client-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063232    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-db\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063246    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-kube-rbac-proxy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063259    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-metrics-client-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063275    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gxx\" (UniqueName: \"kubernetes.io/projected/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-kube-api-access-x2gxx\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063289    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063305    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063320    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.063397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.063335    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-grpc-tls\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.064199 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.064172    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:59.065256 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.065229    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:59.065566 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.065535    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config" (OuterVolumeSpecName: "config") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:59.065635 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.065590    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:59.075206 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.075185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-web-config" (OuterVolumeSpecName: "web-config") pod "fb0eeb5f-c850-47f9-97e9-1528f3c3478f" (UID: "fb0eeb5f-c850-47f9-97e9-1528f3c3478f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:59.163844 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.163818    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.163844 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.163843    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-config\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.163979 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.163856    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.163979 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.163866    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-web-config\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.163979 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.163875    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.163979 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.163884    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.163979 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.163892    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb0eeb5f-c850-47f9-97e9-1528f3c3478f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.628629 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.628592    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerID="e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9" exitCode=0
Sep 10 21:21:59.628826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.628684    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.628826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.628681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerDied","Data":"e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9"}
Sep 10 21:21:59.628826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.628816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb0eeb5f-c850-47f9-97e9-1528f3c3478f","Type":"ContainerDied","Data":"310cba4433885c8afdc4dff3e00335480ea5b385841ac48b6236eefe2075e71a"}
Sep 10 21:21:59.628931 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.628851    2580 scope.go:117] "RemoveContainer" containerID="d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766"
Sep 10 21:21:59.637295 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.637275    2580 scope.go:117] "RemoveContainer" containerID="92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6"
Sep 10 21:21:59.644174 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.644157    2580 scope.go:117] "RemoveContainer" containerID="e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9"
Sep 10 21:21:59.651743 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.651584    2580 scope.go:117] "RemoveContainer" containerID="4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352"
Sep 10 21:21:59.652621 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.652602    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:21:59.656841 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.656810    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:21:59.678688 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.678669    2580 scope.go:117] "RemoveContainer" containerID="af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c"
Sep 10 21:21:59.683931 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.683910    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:21:59.684343 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684322    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="config-reloader"
Sep 10 21:21:59.684386 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684350    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="config-reloader"
Sep 10 21:21:59.684386 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684363    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="thanos-sidecar"
Sep 10 21:21:59.684386 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684373    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="thanos-sidecar"
Sep 10 21:21:59.684386 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684384    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebabd368-7bcb-4047-9e93-88072e7887a4" containerName="registry"
Sep 10 21:21:59.684530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684393    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebabd368-7bcb-4047-9e93-88072e7887a4" containerName="registry"
Sep 10 21:21:59.684530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684438    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy-web"
Sep 10 21:21:59.684530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684449    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy-web"
Sep 10 21:21:59.684530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684460    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy-thanos"
Sep 10 21:21:59.684530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684469    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy-thanos"
Sep 10 21:21:59.684530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684484    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c61a6a-658f-4bd4-aba1-b78bdb0e2513" containerName="console"
Sep 10 21:21:59.684530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684493    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c61a6a-658f-4bd4-aba1-b78bdb0e2513" containerName="console"
Sep 10 21:21:59.684530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684503    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy"
Sep 10 21:21:59.684530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684511    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy"
Sep 10 21:21:59.684530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684522    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="init-config-reloader"
Sep 10 21:21:59.684530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684531    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="init-config-reloader"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684541    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="prometheus"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684549    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="prometheus"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684566    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="315ce57e-6d5a-41a7-860e-f629c4f3e5db" containerName="console"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684575    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="315ce57e-6d5a-41a7-860e-f629c4f3e5db" containerName="console"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684652    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="thanos-sidecar"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684667    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="315ce57e-6d5a-41a7-860e-f629c4f3e5db" containerName="console"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684675    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684685    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="config-reloader"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684695    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c61a6a-658f-4bd4-aba1-b78bdb0e2513" containerName="console"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684702    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy-web"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684712    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="prometheus"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684721    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" containerName="kube-rbac-proxy-thanos"
Sep 10 21:21:59.684825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.684729    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebabd368-7bcb-4047-9e93-88072e7887a4" containerName="registry"
Sep 10 21:21:59.687237 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.687220    2580 scope.go:117] "RemoveContainer" containerID="c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d"
Sep 10 21:21:59.689143 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.689123    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.697079 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697051    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 21:21:59.697304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697291    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 21:21:59.697374 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697356    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 21:21:59.697578 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697483    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 21:21:59.697578 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697495    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 21:21:59.697698 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697670    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 21:21:59.697698 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697684    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3td07tp6mdq28\""
Sep 10 21:21:59.697806 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697761    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 21:21:59.697806 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697770    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 21:21:59.697806 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697786    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gc7bc\""
Sep 10 21:21:59.697806 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697772    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 21:21:59.697997 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697812    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 21:21:59.697997 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.697904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 21:21:59.698139 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.698110    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 21:21:59.701909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.701833    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 21:21:59.704240 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.704221    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 21:21:59.705578 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.705509    2580 scope.go:117] "RemoveContainer" containerID="5ee6a605c85b3c189af497b84ba7a7df80c62fbf06e9d375c918125a7d81166c"
Sep 10 21:21:59.712839 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.712812    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:21:59.712901 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.712884    2580 scope.go:117] "RemoveContainer" containerID="d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766"
Sep 10 21:21:59.713178 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:21:59.713159    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766\": container with ID starting with d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766 not found: ID does not exist" containerID="d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766"
Sep 10 21:21:59.713250 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.713189    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766"} err="failed to get container status \"d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766\": rpc error: code = NotFound desc = could not find container \"d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766\": container with ID starting with d92a7fe6709c591cf57826c5bb92f35f2a2671f577d6757a9f87531580f83766 not found: ID does not exist"
Sep 10 21:21:59.713250 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.713213    2580 scope.go:117] "RemoveContainer" containerID="92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6"
Sep 10 21:21:59.713509 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:21:59.713487    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6\": container with ID starting with 92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6 not found: ID does not exist" containerID="92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6"
Sep 10 21:21:59.713554 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.713522    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6"} err="failed to get container status \"92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6\": rpc error: code = NotFound desc = could not find container \"92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6\": container with ID starting with 92b4659f678c315d000dc7e49fe92ddea343697a1c7e885abbc6f3cfdbc2fcf6 not found: ID does not exist"
Sep 10 21:21:59.713554 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.713545    2580 scope.go:117] "RemoveContainer" containerID="e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9"
Sep 10 21:21:59.713821 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:21:59.713794    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9\": container with ID starting with e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9 not found: ID does not exist" containerID="e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9"
Sep 10 21:21:59.713919 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.713823    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9"} err="failed to get container status \"e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9\": rpc error: code = NotFound desc = could not find container \"e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9\": container with ID starting with e03382077c7a48348544fe4d3e7637ad19542d27368f6d86ae543c0d611cecc9 not found: ID does not exist"
Sep 10 21:21:59.713919 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.713842    2580 scope.go:117] "RemoveContainer" containerID="4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352"
Sep 10 21:21:59.714172 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:21:59.714144    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352\": container with ID starting with 4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352 not found: ID does not exist" containerID="4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352"
Sep 10 21:21:59.714258 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.714179    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352"} err="failed to get container status \"4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352\": rpc error: code = NotFound desc = could not find container \"4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352\": container with ID starting with 4c1373c2eab13db798212dab921d1625f933f56cb9addeec79e4a45cc4a7a352 not found: ID does not exist"
Sep 10 21:21:59.714258 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.714199    2580 scope.go:117] "RemoveContainer" containerID="af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c"
Sep 10 21:21:59.714564 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:21:59.714541    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c\": container with ID starting with af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c not found: ID does not exist" containerID="af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c"
Sep 10 21:21:59.714710 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.714567    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c"} err="failed to get container status \"af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c\": rpc error: code = NotFound desc = could not find container \"af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c\": container with ID starting with af6e9ec41a245626342b1274781e67dda7a35044fef9a021dca12fdd2bb2f83c not found: ID does not exist"
Sep 10 21:21:59.714710 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.714589    2580 scope.go:117] "RemoveContainer" containerID="c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d"
Sep 10 21:21:59.714885 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:21:59.714867    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d\": container with ID starting with c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d not found: ID does not exist" containerID="c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d"
Sep 10 21:21:59.714951 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.714893    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d"} err="failed to get container status \"c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d\": rpc error: code = NotFound desc = could not find container \"c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d\": container with ID starting with c67ee58916ea22c0bb8d67fab88f34805c99b678f317f120cf853dc5b763ee8d not found: ID does not exist"
Sep 10 21:21:59.714951 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.714915    2580 scope.go:117] "RemoveContainer" containerID="5ee6a605c85b3c189af497b84ba7a7df80c62fbf06e9d375c918125a7d81166c"
Sep 10 21:21:59.715200 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:21:59.715176    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee6a605c85b3c189af497b84ba7a7df80c62fbf06e9d375c918125a7d81166c\": container with ID starting with 5ee6a605c85b3c189af497b84ba7a7df80c62fbf06e9d375c918125a7d81166c not found: ID does not exist" containerID="5ee6a605c85b3c189af497b84ba7a7df80c62fbf06e9d375c918125a7d81166c"
Sep 10 21:21:59.715255 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.715207    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee6a605c85b3c189af497b84ba7a7df80c62fbf06e9d375c918125a7d81166c"} err="failed to get container status \"5ee6a605c85b3c189af497b84ba7a7df80c62fbf06e9d375c918125a7d81166c\": rpc error: code = NotFound desc = could not find container \"5ee6a605c85b3c189af497b84ba7a7df80c62fbf06e9d375c918125a7d81166c\": container with ID starting with 5ee6a605c85b3c189af497b84ba7a7df80c62fbf06e9d375c918125a7d81166c not found: ID does not exist"
Sep 10 21:21:59.767655 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.767787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.767787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn26f\" (UniqueName: \"kubernetes.io/projected/f7c7dd78-3bee-420f-820e-5c433e7c4c54-kube-api-access-fn26f\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.767787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-config\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.767787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.767992 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7c7dd78-3bee-420f-820e-5c433e7c4c54-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.767992 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.767992 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.767992 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.767992 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.767992 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.767992 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.767985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.768231 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.768016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.768231 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.768032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.768231 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.768050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.768231 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.768067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.768231 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.768090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7c7dd78-3bee-420f-820e-5c433e7c4c54-config-out\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.768231 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.768132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.768231 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.768152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-web-config\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.768231 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.768171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.818600 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.818579    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb0eeb5f-c850-47f9-97e9-1528f3c3478f" path="/var/lib/kubelet/pods/fb0eeb5f-c850-47f9-97e9-1528f3c3478f/volumes"
Sep 10 21:21:59.868678 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.868652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.868984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.868684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.868984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.868699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.868984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.868731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.868984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.868756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.868984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.868865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.868984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.868890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.868984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.868920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.868984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.868948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.868984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.868980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7c7dd78-3bee-420f-820e-5c433e7c4c54-config-out\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.869524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.869009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.869524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.869033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-web-config\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.869524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.869059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.869524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.869118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.869524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.869144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.869524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.869172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn26f\" (UniqueName: \"kubernetes.io/projected/f7c7dd78-3bee-420f-820e-5c433e7c4c54-kube-api-access-fn26f\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.869524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.869203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-config\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.869524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.869230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.869524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.869266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7c7dd78-3bee-420f-820e-5c433e7c4c54-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.869524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.869294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.870180 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.869887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.870312 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.870247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.871766 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.870697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.871766 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.871650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.872247 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.872204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.872247 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.872217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.874185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.872513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.874185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.873007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.874185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.873107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.874185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.873719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.875312 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.875288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.879801 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.875486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7c7dd78-3bee-420f-820e-5c433e7c4c54-config-out\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.879801 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.875633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-web-config\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.879801 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.875916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.879801 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.875983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7c7dd78-3bee-420f-820e-5c433e7c4c54-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.879801 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.876258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.879801 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.876817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f7c7dd78-3bee-420f-820e-5c433e7c4c54-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.879801 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.877354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.880273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.880139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f7c7dd78-3bee-420f-820e-5c433e7c4c54-config\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:21:59.881384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:21:59.881365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn26f\" (UniqueName: \"kubernetes.io/projected/f7c7dd78-3bee-420f-820e-5c433e7c4c54-kube-api-access-fn26f\") pod \"prometheus-k8s-0\" (UID: \"f7c7dd78-3bee-420f-820e-5c433e7c4c54\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:22:00.008189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:00.008153    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:22:00.298387 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:00.298359    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:22:00.299754 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:22:00.299728    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c7dd78_3bee_420f_820e_5c433e7c4c54.slice/crio-c6bcd321d1ef5bebe25a69013e8825e2f5a850d7ea0993a90752a0ffab271560 WatchSource:0}: Error finding container c6bcd321d1ef5bebe25a69013e8825e2f5a850d7ea0993a90752a0ffab271560: Status 404 returned error can't find the container with id c6bcd321d1ef5bebe25a69013e8825e2f5a850d7ea0993a90752a0ffab271560
Sep 10 21:22:00.636249 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:00.636150    2580 generic.go:356] "Generic (PLEG): container finished" podID="f7c7dd78-3bee-420f-820e-5c433e7c4c54" containerID="dbe735b068c02c1310c31eeda60ad4e77025fc73e9e12e813ed5286a548b52ce" exitCode=0
Sep 10 21:22:00.636249 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:00.636200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7c7dd78-3bee-420f-820e-5c433e7c4c54","Type":"ContainerDied","Data":"dbe735b068c02c1310c31eeda60ad4e77025fc73e9e12e813ed5286a548b52ce"}
Sep 10 21:22:00.636492 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:00.636267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7c7dd78-3bee-420f-820e-5c433e7c4c54","Type":"ContainerStarted","Data":"c6bcd321d1ef5bebe25a69013e8825e2f5a850d7ea0993a90752a0ffab271560"}
Sep 10 21:22:01.644061 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:01.644025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7c7dd78-3bee-420f-820e-5c433e7c4c54","Type":"ContainerStarted","Data":"2e791408cb666f905f3a1e812be7197f8c3215cf708635f0d768e2dfbee7b33f"}
Sep 10 21:22:01.644434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:01.644108    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7c7dd78-3bee-420f-820e-5c433e7c4c54","Type":"ContainerStarted","Data":"063b623d6ef27ac25bef141f9a47a14d5ac01fbebb9b81d4506d8c0d88d0e86a"}
Sep 10 21:22:01.644434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:01.644150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7c7dd78-3bee-420f-820e-5c433e7c4c54","Type":"ContainerStarted","Data":"8ef2fa228ef27958e9fcf41e4aca4efaf0ede2e139c1f2191cc65ef365eb32af"}
Sep 10 21:22:01.644434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:01.644190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7c7dd78-3bee-420f-820e-5c433e7c4c54","Type":"ContainerStarted","Data":"a62b29539c41cb21cedb155b0349b000bb3ff48112eddc5c66bd7ccaa84a8907"}
Sep 10 21:22:01.644434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:01.644228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7c7dd78-3bee-420f-820e-5c433e7c4c54","Type":"ContainerStarted","Data":"f0978b89a060df96f94c3b3a19db4d9e8c6679ecfb1b2be76494cf30ca2abd0c"}
Sep 10 21:22:01.644434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:01.644260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7c7dd78-3bee-420f-820e-5c433e7c4c54","Type":"ContainerStarted","Data":"f612f7272ee8e3c3701d0e94c8c2bd52ae0e153879d9c541847b4d25c6cfe706"}
Sep 10 21:22:01.675646 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:01.675588    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.649324744 podStartE2EDuration="2.675571626s" totalImagesPullingTime="0s" totalInitContainerRuntime="26.246882ms" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:01.673242217 +0000 UTC m=+160.321156384" watchObservedRunningTime="2026-09-10 21:22:01.675571626 +0000 UTC m=+160.323485796"
Sep 10 21:22:05.009217 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:22:05.009173    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:23:00.008525 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:23:00.008476    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:23:00.024735 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:23:00.024707    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:23:00.880058 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:23:00.880030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:24:21.725889 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:21.725861    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 21:24:32.442322 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.442286    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6l6v9"]
Sep 10 21:24:32.444304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.444289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9"
Sep 10 21:24:32.446604 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.446583    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 21:24:32.447298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.447281    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hdrgb\""
Sep 10 21:24:32.447380 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.447304    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 21:24:32.454571 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.454549    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6l6v9"]
Sep 10 21:24:32.607522 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.607465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7a493cd-a6e2-47c6-a116-6f5233772112-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6l6v9\" (UID: \"c7a493cd-a6e2-47c6-a116-6f5233772112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9"
Sep 10 21:24:32.607714 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.607555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mkl\" (UniqueName: \"kubernetes.io/projected/c7a493cd-a6e2-47c6-a116-6f5233772112-kube-api-access-p7mkl\") pod \"cert-manager-webhook-6486ffd997-6l6v9\" (UID: \"c7a493cd-a6e2-47c6-a116-6f5233772112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9"
Sep 10 21:24:32.708683 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.708584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mkl\" (UniqueName: \"kubernetes.io/projected/c7a493cd-a6e2-47c6-a116-6f5233772112-kube-api-access-p7mkl\") pod \"cert-manager-webhook-6486ffd997-6l6v9\" (UID: \"c7a493cd-a6e2-47c6-a116-6f5233772112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9"
Sep 10 21:24:32.708683 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.708682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7a493cd-a6e2-47c6-a116-6f5233772112-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6l6v9\" (UID: \"c7a493cd-a6e2-47c6-a116-6f5233772112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9"
Sep 10 21:24:32.721570 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.721532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7a493cd-a6e2-47c6-a116-6f5233772112-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6l6v9\" (UID: \"c7a493cd-a6e2-47c6-a116-6f5233772112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9"
Sep 10 21:24:32.723572 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.723547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mkl\" (UniqueName: \"kubernetes.io/projected/c7a493cd-a6e2-47c6-a116-6f5233772112-kube-api-access-p7mkl\") pod \"cert-manager-webhook-6486ffd997-6l6v9\" (UID: \"c7a493cd-a6e2-47c6-a116-6f5233772112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9"
Sep 10 21:24:32.759596 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:32.759555    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9"
Sep 10 21:24:33.027333 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:33.027256    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6l6v9"]
Sep 10 21:24:33.030233 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:24:33.030200    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7a493cd_a6e2_47c6_a116_6f5233772112.slice/crio-19a46970fe38ba8ef8e7d593e1cc3a0fb500a4b75a45ba0bae64c32703912728 WatchSource:0}: Error finding container 19a46970fe38ba8ef8e7d593e1cc3a0fb500a4b75a45ba0bae64c32703912728: Status 404 returned error can't find the container with id 19a46970fe38ba8ef8e7d593e1cc3a0fb500a4b75a45ba0bae64c32703912728
Sep 10 21:24:33.032154 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:33.032139    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:24:33.208460 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:33.208431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9" event={"ID":"c7a493cd-a6e2-47c6-a116-6f5233772112","Type":"ContainerStarted","Data":"19a46970fe38ba8ef8e7d593e1cc3a0fb500a4b75a45ba0bae64c32703912728"}
Sep 10 21:24:36.222983 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:36.222947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9" event={"ID":"c7a493cd-a6e2-47c6-a116-6f5233772112","Type":"ContainerStarted","Data":"b24001cc85d090316fefee373d448b7ff00a820e99315890f9d0f7185d25b6e2"}
Sep 10 21:24:36.223452 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:36.223011    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9"
Sep 10 21:24:36.240369 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:36.240316    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9" podStartSLOduration=1.251570185 podStartE2EDuration="4.240299743s" totalImagesPullingTime="2.988729558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:24:36.239107255 +0000 UTC m=+314.887021424" watchObservedRunningTime="2026-09-10 21:24:36.240299743 +0000 UTC m=+314.888213912"
Sep 10 21:24:42.229543 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:42.229510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6l6v9"
Sep 10 21:24:58.783397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.783365    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"]
Sep 10 21:24:58.801887 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.801859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.802358 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.802336    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"]
Sep 10 21:24:58.804792 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.804771    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 21:24:58.805384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.805367    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 21:24:58.805384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.805382    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 21:24:58.805575 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.805542    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jn67t\""
Sep 10 21:24:58.805710 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.805693    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 21:24:58.821096 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.821076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bb49ee-12c5-4d27-821d-62003ccf6724-tmp\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.821184 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.821110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4vft\" (UniqueName: \"kubernetes.io/projected/b2bb49ee-12c5-4d27-821d-62003ccf6724-kube-api-access-f4vft\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.821184 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.821135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2bb49ee-12c5-4d27-821d-62003ccf6724-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.821258 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.821179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2bb49ee-12c5-4d27-821d-62003ccf6724-manifests\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.821258 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.821214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2bb49ee-12c5-4d27-821d-62003ccf6724-webhook-cert\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.922169 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.922140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2bb49ee-12c5-4d27-821d-62003ccf6724-webhook-cert\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.922293 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.922181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bb49ee-12c5-4d27-821d-62003ccf6724-tmp\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.922293 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.922208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4vft\" (UniqueName: \"kubernetes.io/projected/b2bb49ee-12c5-4d27-821d-62003ccf6724-kube-api-access-f4vft\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.922387 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.922286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2bb49ee-12c5-4d27-821d-62003ccf6724-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.922387 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.922351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2bb49ee-12c5-4d27-821d-62003ccf6724-manifests\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.922722 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.922701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2bb49ee-12c5-4d27-821d-62003ccf6724-manifests\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.924616 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.924562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bb49ee-12c5-4d27-821d-62003ccf6724-tmp\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.924810 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.924794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2bb49ee-12c5-4d27-821d-62003ccf6724-webhook-cert\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.924858 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.924840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2bb49ee-12c5-4d27-821d-62003ccf6724-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:58.930728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:58.930705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4vft\" (UniqueName: \"kubernetes.io/projected/b2bb49ee-12c5-4d27-821d-62003ccf6724-kube-api-access-f4vft\") pod \"opendatahub-operator-controller-manager-56fc979fdc-nwwdp\" (UID: \"b2bb49ee-12c5-4d27-821d-62003ccf6724\") " pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:59.112776 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:59.112700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:24:59.406339 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:24:59.406150    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"]
Sep 10 21:24:59.409207 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:24:59.409174    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bb49ee_12c5_4d27_821d_62003ccf6724.slice/crio-d6ba5f9126e35f8d8c2a5165fb4ccef70d30e80a2c28bbf16f90d05588871482 WatchSource:0}: Error finding container d6ba5f9126e35f8d8c2a5165fb4ccef70d30e80a2c28bbf16f90d05588871482: Status 404 returned error can't find the container with id d6ba5f9126e35f8d8c2a5165fb4ccef70d30e80a2c28bbf16f90d05588871482
Sep 10 21:25:00.316570 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:00.316521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp" event={"ID":"b2bb49ee-12c5-4d27-821d-62003ccf6724","Type":"ContainerStarted","Data":"d6ba5f9126e35f8d8c2a5165fb4ccef70d30e80a2c28bbf16f90d05588871482"}
Sep 10 21:25:03.330863 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:03.330822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp" event={"ID":"b2bb49ee-12c5-4d27-821d-62003ccf6724","Type":"ContainerStarted","Data":"b1a900e22b10596cea769a62f406ad11a33afba33b2ab98e73a0d3e4faf779dd"}
Sep 10 21:25:04.336692 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:04.336658    2580 generic.go:356] "Generic (PLEG): container finished" podID="b2bb49ee-12c5-4d27-821d-62003ccf6724" containerID="b1a900e22b10596cea769a62f406ad11a33afba33b2ab98e73a0d3e4faf779dd" exitCode=0
Sep 10 21:25:04.337090 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:04.336747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp" event={"ID":"b2bb49ee-12c5-4d27-821d-62003ccf6724","Type":"ContainerDied","Data":"b1a900e22b10596cea769a62f406ad11a33afba33b2ab98e73a0d3e4faf779dd"}
Sep 10 21:25:05.343181 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:05.343138    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp" event={"ID":"b2bb49ee-12c5-4d27-821d-62003ccf6724","Type":"ContainerStarted","Data":"b3f9888c4e77aa8fe10cf2bbc5a2b0080b39162f873af88ec34bc2109257f03c"}
Sep 10 21:25:05.343606 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:05.343234    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:25:05.365633 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:05.365575    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp" podStartSLOduration=3.041327021 podStartE2EDuration="7.365534749s" totalImagesPullingTime="3.264208102s" totalInitContainerRuntime="1.059999626s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:24:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:25:05.364933347 +0000 UTC m=+344.012847515" watchObservedRunningTime="2026-09-10 21:25:05.365534749 +0000 UTC m=+344.013448917"
Sep 10 21:25:16.286586 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.286499    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"]
Sep 10 21:25:16.290121 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.290098    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.292637 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.292612    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 21:25:16.292637 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.292635    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 21:25:16.293193 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.293174    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 21:25:16.293300 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.293283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 21:25:16.293371 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.293286    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:25:16.293448 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.293376    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-m54tm\""
Sep 10 21:25:16.298101 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.298076    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"]
Sep 10 21:25:16.349960 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.349936    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56fc979fdc-nwwdp"
Sep 10 21:25:16.383548 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.383516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b27662b-ebb7-4c4a-89db-39d6a85d017e-metrics-cert\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.383710 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.383630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b27662b-ebb7-4c4a-89db-39d6a85d017e-cert\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.383710 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.383682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7dhw\" (UniqueName: \"kubernetes.io/projected/7b27662b-ebb7-4c4a-89db-39d6a85d017e-kube-api-access-l7dhw\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.383822 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.383767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7b27662b-ebb7-4c4a-89db-39d6a85d017e-manager-config\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.484567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.484520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dhw\" (UniqueName: \"kubernetes.io/projected/7b27662b-ebb7-4c4a-89db-39d6a85d017e-kube-api-access-l7dhw\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.484769 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.484607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7b27662b-ebb7-4c4a-89db-39d6a85d017e-manager-config\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.484769 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.484635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b27662b-ebb7-4c4a-89db-39d6a85d017e-metrics-cert\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.484769 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.484686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b27662b-ebb7-4c4a-89db-39d6a85d017e-cert\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.485453 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.485365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7b27662b-ebb7-4c4a-89db-39d6a85d017e-manager-config\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.487254 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.487233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b27662b-ebb7-4c4a-89db-39d6a85d017e-metrics-cert\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.487344 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.487259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b27662b-ebb7-4c4a-89db-39d6a85d017e-cert\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.492214 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.492195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7dhw\" (UniqueName: \"kubernetes.io/projected/7b27662b-ebb7-4c4a-89db-39d6a85d017e-kube-api-access-l7dhw\") pod \"lws-controller-manager-5644d67b8d-24f8x\" (UID: \"7b27662b-ebb7-4c4a-89db-39d6a85d017e\") " pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.600311 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.600230    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:16.877489 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:16.877381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"]
Sep 10 21:25:16.880108 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:25:16.880078    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b27662b_ebb7_4c4a_89db_39d6a85d017e.slice/crio-e566584cfd052b112012dcbf72875586a58981bfd70ffd8a32d5f3440e9dc33a WatchSource:0}: Error finding container e566584cfd052b112012dcbf72875586a58981bfd70ffd8a32d5f3440e9dc33a: Status 404 returned error can't find the container with id e566584cfd052b112012dcbf72875586a58981bfd70ffd8a32d5f3440e9dc33a
Sep 10 21:25:17.396165 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:17.396128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x" event={"ID":"7b27662b-ebb7-4c4a-89db-39d6a85d017e","Type":"ContainerStarted","Data":"e566584cfd052b112012dcbf72875586a58981bfd70ffd8a32d5f3440e9dc33a"}
Sep 10 21:25:18.577342 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.577305    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-745b97c954-n22zc"]
Sep 10 21:25:18.584379 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.584346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:18.589387 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.589364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 21:25:18.590099 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.590062    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 21:25:18.590287 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.590257    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 21:25:18.592781 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.592759    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 21:25:18.596593 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.596574    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2l9bn\""
Sep 10 21:25:18.603497 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.603478    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-745b97c954-n22zc"]
Sep 10 21:25:18.702546 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.702507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdb78\" (UniqueName: \"kubernetes.io/projected/71ccef02-ed65-4e34-ac84-65ee182041d4-kube-api-access-pdb78\") pod \"kube-auth-proxy-745b97c954-n22zc\" (UID: \"71ccef02-ed65-4e34-ac84-65ee182041d4\") " pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:18.702722 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.702569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ccef02-ed65-4e34-ac84-65ee182041d4-tls-certs\") pod \"kube-auth-proxy-745b97c954-n22zc\" (UID: \"71ccef02-ed65-4e34-ac84-65ee182041d4\") " pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:18.702722 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.702622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71ccef02-ed65-4e34-ac84-65ee182041d4-tmp\") pod \"kube-auth-proxy-745b97c954-n22zc\" (UID: \"71ccef02-ed65-4e34-ac84-65ee182041d4\") " pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:18.803326 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.803291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71ccef02-ed65-4e34-ac84-65ee182041d4-tmp\") pod \"kube-auth-proxy-745b97c954-n22zc\" (UID: \"71ccef02-ed65-4e34-ac84-65ee182041d4\") " pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:18.803542 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.803394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdb78\" (UniqueName: \"kubernetes.io/projected/71ccef02-ed65-4e34-ac84-65ee182041d4-kube-api-access-pdb78\") pod \"kube-auth-proxy-745b97c954-n22zc\" (UID: \"71ccef02-ed65-4e34-ac84-65ee182041d4\") " pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:18.803542 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.803471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ccef02-ed65-4e34-ac84-65ee182041d4-tls-certs\") pod \"kube-auth-proxy-745b97c954-n22zc\" (UID: \"71ccef02-ed65-4e34-ac84-65ee182041d4\") " pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:18.803662 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:25:18.803602    2580 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 21:25:18.803712 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:25:18.803665    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71ccef02-ed65-4e34-ac84-65ee182041d4-tls-certs podName:71ccef02-ed65-4e34-ac84-65ee182041d4 nodeName:}" failed. No retries permitted until 2026-09-10 21:25:19.303645748 +0000 UTC m=+357.951559934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/71ccef02-ed65-4e34-ac84-65ee182041d4-tls-certs") pod "kube-auth-proxy-745b97c954-n22zc" (UID: "71ccef02-ed65-4e34-ac84-65ee182041d4") : secret "kube-auth-proxy-tls" not found
Sep 10 21:25:18.806037 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.806009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71ccef02-ed65-4e34-ac84-65ee182041d4-tmp\") pod \"kube-auth-proxy-745b97c954-n22zc\" (UID: \"71ccef02-ed65-4e34-ac84-65ee182041d4\") " pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:18.815934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:18.815900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdb78\" (UniqueName: \"kubernetes.io/projected/71ccef02-ed65-4e34-ac84-65ee182041d4-kube-api-access-pdb78\") pod \"kube-auth-proxy-745b97c954-n22zc\" (UID: \"71ccef02-ed65-4e34-ac84-65ee182041d4\") " pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:19.309049 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:19.309013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ccef02-ed65-4e34-ac84-65ee182041d4-tls-certs\") pod \"kube-auth-proxy-745b97c954-n22zc\" (UID: \"71ccef02-ed65-4e34-ac84-65ee182041d4\") " pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:19.311799 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:19.311770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ccef02-ed65-4e34-ac84-65ee182041d4-tls-certs\") pod \"kube-auth-proxy-745b97c954-n22zc\" (UID: \"71ccef02-ed65-4e34-ac84-65ee182041d4\") " pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:19.495636 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:19.495602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc"
Sep 10 21:25:19.781321 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:19.781294    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-745b97c954-n22zc"]
Sep 10 21:25:19.783622 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:25:19.783591    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ccef02_ed65_4e34_ac84_65ee182041d4.slice/crio-793fc0767459c0904fb2f9f92fae15895072fb084f2d0f99865f8ae0fc5dc598 WatchSource:0}: Error finding container 793fc0767459c0904fb2f9f92fae15895072fb084f2d0f99865f8ae0fc5dc598: Status 404 returned error can't find the container with id 793fc0767459c0904fb2f9f92fae15895072fb084f2d0f99865f8ae0fc5dc598
Sep 10 21:25:20.412678 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:20.412635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x" event={"ID":"7b27662b-ebb7-4c4a-89db-39d6a85d017e","Type":"ContainerStarted","Data":"ec55d8e5362735c01b51a1fba4e8b5acd4990381c50438206e8689307ed0fd5e"}
Sep 10 21:25:20.412858 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:20.412763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:25:20.414252 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:20.414226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc" event={"ID":"71ccef02-ed65-4e34-ac84-65ee182041d4","Type":"ContainerStarted","Data":"793fc0767459c0904fb2f9f92fae15895072fb084f2d0f99865f8ae0fc5dc598"}
Sep 10 21:25:20.433763 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:20.433712    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x" podStartSLOduration=1.811298625 podStartE2EDuration="4.433690795s" totalImagesPullingTime="2.62239217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:25:20.43196253 +0000 UTC m=+359.079876698" watchObservedRunningTime="2026-09-10 21:25:20.433690795 +0000 UTC m=+359.081604963"
Sep 10 21:25:22.690785 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:22.690765    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 21:25:23.429067 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:23.429032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc" event={"ID":"71ccef02-ed65-4e34-ac84-65ee182041d4","Type":"ContainerStarted","Data":"3c866cacd6a9c246d2c5e014d9a4f27c71a2bf9b12e9208833176877e9e9ae9d"}
Sep 10 21:25:23.445603 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:23.445553    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-745b97c954-n22zc" podStartSLOduration=2.54251198 podStartE2EDuration="5.445537623s" totalImagesPullingTime="2.903025643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:25:23.444989617 +0000 UTC m=+362.092903786" watchObservedRunningTime="2026-09-10 21:25:23.445537623 +0000 UTC m=+362.093451803"
Sep 10 21:25:31.421922 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:25:31.421885    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5644d67b8d-24f8x"
Sep 10 21:27:10.805607 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:10.805569    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"]
Sep 10 21:27:10.808719 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:10.808702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:10.811259 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:10.811235    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 21:27:10.811399 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:10.811282    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 21:27:10.811399 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:10.811290    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-99snq\""
Sep 10 21:27:10.824696 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:10.824670    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"]
Sep 10 21:27:10.971836 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:10.971793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv28z\" (UniqueName: \"kubernetes.io/projected/979683d7-4a6e-423b-b240-14de67489ac4-kube-api-access-hv28z\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wphz9\" (UID: \"979683d7-4a6e-423b-b240-14de67489ac4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:10.972015 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:10.971857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/979683d7-4a6e-423b-b240-14de67489ac4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wphz9\" (UID: \"979683d7-4a6e-423b-b240-14de67489ac4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:10.998200 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:10.998162    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"]
Sep 10 21:27:10.998479 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:27:10.998458    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-hv28z], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9" podUID="979683d7-4a6e-423b-b240-14de67489ac4"
Sep 10 21:27:11.008692 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.008649    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"]
Sep 10 21:27:11.072708 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.072625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv28z\" (UniqueName: \"kubernetes.io/projected/979683d7-4a6e-423b-b240-14de67489ac4-kube-api-access-hv28z\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wphz9\" (UID: \"979683d7-4a6e-423b-b240-14de67489ac4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:11.072862 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.072747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/979683d7-4a6e-423b-b240-14de67489ac4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wphz9\" (UID: \"979683d7-4a6e-423b-b240-14de67489ac4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:11.073123 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.073100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/979683d7-4a6e-423b-b240-14de67489ac4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wphz9\" (UID: \"979683d7-4a6e-423b-b240-14de67489ac4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:11.076016 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:27:11.075991    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hv28z for pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-135-226.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-135-226.ec2.internal' and this object
Sep 10 21:27:11.076125 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:27:11.076098    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/979683d7-4a6e-423b-b240-14de67489ac4-kube-api-access-hv28z podName:979683d7-4a6e-423b-b240-14de67489ac4 nodeName:}" failed. No retries permitted until 2026-09-10 21:27:11.576076224 +0000 UTC m=+470.223990383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hv28z" (UniqueName: "kubernetes.io/projected/979683d7-4a6e-423b-b240-14de67489ac4-kube-api-access-hv28z") pod "kuadrant-operator-controller-manager-7978cbdfbc-wphz9" (UID: "979683d7-4a6e-423b-b240-14de67489ac4") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-135-226.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-135-226.ec2.internal' and this object
Sep 10 21:27:11.098842 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.098809    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk"]
Sep 10 21:27:11.102075 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.102059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk"
Sep 10 21:27:11.106454 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.106432    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tgj2c\""
Sep 10 21:27:11.137088 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.137060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk"]
Sep 10 21:27:11.274613 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.274583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg455\" (UniqueName: \"kubernetes.io/projected/c34ffe9d-afe1-4d32-bf40-3f4eacdd0423-kube-api-access-rg455\") pod \"limitador-operator-controller-manager-74495c69f8-t2gmk\" (UID: \"c34ffe9d-afe1-4d32-bf40-3f4eacdd0423\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk"
Sep 10 21:27:11.375609 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.375526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg455\" (UniqueName: \"kubernetes.io/projected/c34ffe9d-afe1-4d32-bf40-3f4eacdd0423-kube-api-access-rg455\") pod \"limitador-operator-controller-manager-74495c69f8-t2gmk\" (UID: \"c34ffe9d-afe1-4d32-bf40-3f4eacdd0423\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk"
Sep 10 21:27:11.394303 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.394280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg455\" (UniqueName: \"kubernetes.io/projected/c34ffe9d-afe1-4d32-bf40-3f4eacdd0423-kube-api-access-rg455\") pod \"limitador-operator-controller-manager-74495c69f8-t2gmk\" (UID: \"c34ffe9d-afe1-4d32-bf40-3f4eacdd0423\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk"
Sep 10 21:27:11.413163 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.413133    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk"
Sep 10 21:27:11.577922 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.577883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv28z\" (UniqueName: \"kubernetes.io/projected/979683d7-4a6e-423b-b240-14de67489ac4-kube-api-access-hv28z\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wphz9\" (UID: \"979683d7-4a6e-423b-b240-14de67489ac4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:11.580530 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:27:11.580508    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hv28z for pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-135-226.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-135-226.ec2.internal' and this object
Sep 10 21:27:11.580612 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:27:11.580574    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/979683d7-4a6e-423b-b240-14de67489ac4-kube-api-access-hv28z podName:979683d7-4a6e-423b-b240-14de67489ac4 nodeName:}" failed. No retries permitted until 2026-09-10 21:27:12.580558806 +0000 UTC m=+471.228472955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hv28z" (UniqueName: "kubernetes.io/projected/979683d7-4a6e-423b-b240-14de67489ac4-kube-api-access-hv28z") pod "kuadrant-operator-controller-manager-7978cbdfbc-wphz9" (UID: "979683d7-4a6e-423b-b240-14de67489ac4") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-135-226.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-135-226.ec2.internal' and this object
Sep 10 21:27:11.695195 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.695166    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk"]
Sep 10 21:27:11.698001 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:27:11.697972    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34ffe9d_afe1_4d32_bf40_3f4eacdd0423.slice/crio-905d120536acf7a81eaee57f46e6963ae28d6f85333f39bdd3b45b3a886c71e8 WatchSource:0}: Error finding container 905d120536acf7a81eaee57f46e6963ae28d6f85333f39bdd3b45b3a886c71e8: Status 404 returned error can't find the container with id 905d120536acf7a81eaee57f46e6963ae28d6f85333f39bdd3b45b3a886c71e8
Sep 10 21:27:11.888917 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.888884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk" event={"ID":"c34ffe9d-afe1-4d32-bf40-3f4eacdd0423","Type":"ContainerStarted","Data":"905d120536acf7a81eaee57f46e6963ae28d6f85333f39bdd3b45b3a886c71e8"}
Sep 10 21:27:11.889370 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.888897    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:11.897619 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.897589    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:11.899933 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:27:11.899909    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-wphz9\" is forbidden: User \"system:node:ip-10-0-135-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-226.ec2.internal' and this object" podUID="979683d7-4a6e-423b-b240-14de67489ac4" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:11.981369 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.981293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/979683d7-4a6e-423b-b240-14de67489ac4-extensions-socket-volume\") pod \"979683d7-4a6e-423b-b240-14de67489ac4\" (UID: \"979683d7-4a6e-423b-b240-14de67489ac4\") "
Sep 10 21:27:11.981522 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.981509    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv28z\" (UniqueName: \"kubernetes.io/projected/979683d7-4a6e-423b-b240-14de67489ac4-kube-api-access-hv28z\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:27:11.981572 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:11.981553    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979683d7-4a6e-423b-b240-14de67489ac4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "979683d7-4a6e-423b-b240-14de67489ac4" (UID: "979683d7-4a6e-423b-b240-14de67489ac4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:27:12.082474 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:12.082444    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/979683d7-4a6e-423b-b240-14de67489ac4-extensions-socket-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:27:12.893894 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:12.893856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:12.896083 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:27:12.896046    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-wphz9\" is forbidden: User \"system:node:ip-10-0-135-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-226.ec2.internal' and this object" podUID="979683d7-4a6e-423b-b240-14de67489ac4" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:12.902270 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:27:12.902234    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-wphz9\" is forbidden: User \"system:node:ip-10-0-135-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-226.ec2.internal' and this object" podUID="979683d7-4a6e-423b-b240-14de67489ac4" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wphz9"
Sep 10 21:27:13.821183 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:13.821135    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="979683d7-4a6e-423b-b240-14de67489ac4" path="/var/lib/kubelet/pods/979683d7-4a6e-423b-b240-14de67489ac4/volumes"
Sep 10 21:27:14.905032 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:14.904994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk" event={"ID":"c34ffe9d-afe1-4d32-bf40-3f4eacdd0423","Type":"ContainerStarted","Data":"10b1e4b9793d9d5cbb7dd2d8b95f5ce0d5e48b5bb79b8bc34047539341a1a546"}
Sep 10 21:27:14.905525 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:14.905142    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk"
Sep 10 21:27:14.928710 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:14.928657    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk" podStartSLOduration=1.594541829 podStartE2EDuration="3.928641286s" totalImagesPullingTime="2.334099457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:27:14.926848418 +0000 UTC m=+473.574762585" watchObservedRunningTime="2026-09-10 21:27:14.928641286 +0000 UTC m=+473.576555453"
Sep 10 21:27:25.911847 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:25.911812    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t2gmk"
Sep 10 21:27:45.483597 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.483557    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:27:45.485698 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.485679    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz"
Sep 10 21:27:45.488106 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.488081    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 21:27:45.488273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.488257    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7kgdq\""
Sep 10 21:27:45.496716 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.496680    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:27:45.529303 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.529270    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:27:45.557213 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.557174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5f352c56-671d-4097-b21e-cc931ff7398e-config-file\") pod \"limitador-limitador-84994f7ffc-z5npz\" (UID: \"5f352c56-671d-4097-b21e-cc931ff7398e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz"
Sep 10 21:27:45.557393 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.557242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj7l7\" (UniqueName: \"kubernetes.io/projected/5f352c56-671d-4097-b21e-cc931ff7398e-kube-api-access-gj7l7\") pod \"limitador-limitador-84994f7ffc-z5npz\" (UID: \"5f352c56-671d-4097-b21e-cc931ff7398e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz"
Sep 10 21:27:45.658509 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.658471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj7l7\" (UniqueName: \"kubernetes.io/projected/5f352c56-671d-4097-b21e-cc931ff7398e-kube-api-access-gj7l7\") pod \"limitador-limitador-84994f7ffc-z5npz\" (UID: \"5f352c56-671d-4097-b21e-cc931ff7398e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz"
Sep 10 21:27:45.658675 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.658561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5f352c56-671d-4097-b21e-cc931ff7398e-config-file\") pod \"limitador-limitador-84994f7ffc-z5npz\" (UID: \"5f352c56-671d-4097-b21e-cc931ff7398e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz"
Sep 10 21:27:45.659136 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.659118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5f352c56-671d-4097-b21e-cc931ff7398e-config-file\") pod \"limitador-limitador-84994f7ffc-z5npz\" (UID: \"5f352c56-671d-4097-b21e-cc931ff7398e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz"
Sep 10 21:27:45.667100 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.667074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj7l7\" (UniqueName: \"kubernetes.io/projected/5f352c56-671d-4097-b21e-cc931ff7398e-kube-api-access-gj7l7\") pod \"limitador-limitador-84994f7ffc-z5npz\" (UID: \"5f352c56-671d-4097-b21e-cc931ff7398e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz"
Sep 10 21:27:45.797502 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:45.797395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz"
Sep 10 21:27:46.075879 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:46.075763    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:27:46.078429 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:27:46.078387    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f352c56_671d_4097_b21e_cc931ff7398e.slice/crio-eafd22c0b8ddda53344762fd4ecb158dfc1b8fa1e90a15a88d113d9cd3532f56 WatchSource:0}: Error finding container eafd22c0b8ddda53344762fd4ecb158dfc1b8fa1e90a15a88d113d9cd3532f56: Status 404 returned error can't find the container with id eafd22c0b8ddda53344762fd4ecb158dfc1b8fa1e90a15a88d113d9cd3532f56
Sep 10 21:27:47.044722 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:47.044679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz" event={"ID":"5f352c56-671d-4097-b21e-cc931ff7398e","Type":"ContainerStarted","Data":"eafd22c0b8ddda53344762fd4ecb158dfc1b8fa1e90a15a88d113d9cd3532f56"}
Sep 10 21:27:48.050630 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:48.050597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz" event={"ID":"5f352c56-671d-4097-b21e-cc931ff7398e","Type":"ContainerStarted","Data":"dfb8a373530272495c05f5b8506c9183e4535a2c36721b322688efffc942b172"}
Sep 10 21:27:48.051071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:48.050705    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz"
Sep 10 21:27:48.076286 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:48.068602    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz" podStartSLOduration=1.776103198 podStartE2EDuration="3.06858377s" totalImagesPullingTime="1.292480572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:27:48.066570427 +0000 UTC m=+506.714484594" watchObservedRunningTime="2026-09-10 21:27:48.06858377 +0000 UTC m=+506.716497939"
Sep 10 21:27:59.055947 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:27:59.055913    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-z5npz"
Sep 10 21:28:54.224103 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:28:54.224067    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:29:12.707666 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:29:12.707629    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:29:34.703281 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:29:34.703245    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:29:40.307818 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:29:40.307772    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:29:47.313537 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:29:47.313495    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:30:00.139857 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:00.139821    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-fb2dg"]
Sep 10 21:30:00.142261 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:00.142238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-fb2dg"
Sep 10 21:30:00.144615 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:00.144597    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wg7l\""
Sep 10 21:30:00.150573 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:00.150099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-fb2dg"]
Sep 10 21:30:00.153246 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:00.153223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-fb2dg"
Sep 10 21:30:00.430821 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:00.430746    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-fb2dg"]
Sep 10 21:30:00.433941 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:30:00.433914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a5957e_df05_46d9_ad32_79150fca3c8f.slice/crio-47dfd425f2196291ed8e06d4b328170b788bb634c917123faac39cb31ce578e0 WatchSource:0}: Error finding container 47dfd425f2196291ed8e06d4b328170b788bb634c917123faac39cb31ce578e0: Status 404 returned error can't find the container with id 47dfd425f2196291ed8e06d4b328170b788bb634c917123faac39cb31ce578e0
Sep 10 21:30:00.436276 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:00.436256    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:30:00.629614 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:00.629576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-fb2dg" event={"ID":"76a5957e-df05-46d9-ad32-79150fca3c8f","Type":"ContainerStarted","Data":"47dfd425f2196291ed8e06d4b328170b788bb634c917123faac39cb31ce578e0"}
Sep 10 21:30:03.644656 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:03.644571    2580 generic.go:356] "Generic (PLEG): container finished" podID="76a5957e-df05-46d9-ad32-79150fca3c8f" containerID="8328af01c59988bc9689d1fd1748499242bdbbfd4738df6e4527e626095600df" exitCode=0
Sep 10 21:30:03.645114 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:03.644654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-fb2dg" event={"ID":"76a5957e-df05-46d9-ad32-79150fca3c8f","Type":"ContainerDied","Data":"8328af01c59988bc9689d1fd1748499242bdbbfd4738df6e4527e626095600df"}
Sep 10 21:30:03.709272 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:03.709238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:30:04.768756 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:04.768735    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-fb2dg"
Sep 10 21:30:05.654956 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:05.654931    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-fb2dg"
Sep 10 21:30:05.655142 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:05.654930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-fb2dg" event={"ID":"76a5957e-df05-46d9-ad32-79150fca3c8f","Type":"ContainerDied","Data":"47dfd425f2196291ed8e06d4b328170b788bb634c917123faac39cb31ce578e0"}
Sep 10 21:30:05.655142 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:05.655079    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47dfd425f2196291ed8e06d4b328170b788bb634c917123faac39cb31ce578e0"
Sep 10 21:30:23.614283 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.614248    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"]
Sep 10 21:30:23.614724 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.614609    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a5957e-df05-46d9-ad32-79150fca3c8f" containerName="cleanup"
Sep 10 21:30:23.614724 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.614623    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a5957e-df05-46d9-ad32-79150fca3c8f" containerName="cleanup"
Sep 10 21:30:23.614724 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.614703    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a5957e-df05-46d9-ad32-79150fca3c8f" containerName="cleanup"
Sep 10 21:30:23.622086 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.622062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.624475 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.624452    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 21:30:23.625232 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.625209    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 21:30:23.625332 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.625268    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 21:30:23.625332 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.625290    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dxtxk\""
Sep 10 21:30:23.630101 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.629937    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"]
Sep 10 21:30:23.709621 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.709577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.709826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.709656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.709826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.709679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqcxj\" (UniqueName: \"kubernetes.io/projected/a4592aec-60bd-4e40-8b74-c38fa6c963b9-kube-api-access-kqcxj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.709826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.709733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.709826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.709775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.709964 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.709837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4592aec-60bd-4e40-8b74-c38fa6c963b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.811170 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.811142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.811170 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.811173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.811397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.811196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4592aec-60bd-4e40-8b74-c38fa6c963b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.811397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.811229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.811397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.811280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.811397 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.811302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqcxj\" (UniqueName: \"kubernetes.io/projected/a4592aec-60bd-4e40-8b74-c38fa6c963b9-kube-api-access-kqcxj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.811708 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.811684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.811772 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.811738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.811816 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.811744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.813614 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.813592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4592aec-60bd-4e40-8b74-c38fa6c963b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.813767 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.813749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4592aec-60bd-4e40-8b74-c38fa6c963b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.818821 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.818801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqcxj\" (UniqueName: \"kubernetes.io/projected/a4592aec-60bd-4e40-8b74-c38fa6c963b9-kube-api-access-kqcxj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-klq8j\" (UID: \"a4592aec-60bd-4e40-8b74-c38fa6c963b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:23.934520 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:23.934456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:24.205286 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:24.205158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"]
Sep 10 21:30:24.208009 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:30:24.207983    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4592aec_60bd_4e40_8b74_c38fa6c963b9.slice/crio-cb70c24261b70e1e6d82a7efdf071d2446b69043c0321145d8b1742c96554f21 WatchSource:0}: Error finding container cb70c24261b70e1e6d82a7efdf071d2446b69043c0321145d8b1742c96554f21: Status 404 returned error can't find the container with id cb70c24261b70e1e6d82a7efdf071d2446b69043c0321145d8b1742c96554f21
Sep 10 21:30:24.739748 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:24.739709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j" event={"ID":"a4592aec-60bd-4e40-8b74-c38fa6c963b9","Type":"ContainerStarted","Data":"cb70c24261b70e1e6d82a7efdf071d2446b69043c0321145d8b1742c96554f21"}
Sep 10 21:30:25.514544 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:25.514508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:30:25.745442 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:25.745393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j" event={"ID":"a4592aec-60bd-4e40-8b74-c38fa6c963b9","Type":"ContainerStarted","Data":"efb1cd8e1ac8e421c480da21f9e335eb3ae42bede6763d27eb3413ed6762d192"}
Sep 10 21:30:25.766362 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:25.766310    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j" podStartSLOduration=1.635822243 podStartE2EDuration="2.76629373s" totalImagesPullingTime="1.130471487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:30:25.765191724 +0000 UTC m=+664.413105901" watchObservedRunningTime="2026-09-10 21:30:25.76629373 +0000 UTC m=+664.414207897"
Sep 10 21:30:30.009617 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.009588    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"]
Sep 10 21:30:30.013240 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.013216    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.015797 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.015776    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 21:30:30.025537 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.025508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"]
Sep 10 21:30:30.167034 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.166995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.167205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.167096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.167205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.167134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58532a1e-d1d4-43e3-b156-349d7950668c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.167205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.167151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn9d4\" (UniqueName: \"kubernetes.io/projected/58532a1e-d1d4-43e3-b156-349d7950668c-kube-api-access-jn9d4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.167205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.167173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.167205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.167199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.268630 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.268541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.268630 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.268587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58532a1e-d1d4-43e3-b156-349d7950668c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.268630 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.268606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9d4\" (UniqueName: \"kubernetes.io/projected/58532a1e-d1d4-43e3-b156-349d7950668c-kube-api-access-jn9d4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.268630 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.268630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.268893 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.268811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.268893 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.268866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.269086 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.269063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.269161 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.269136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.269213 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.269162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.270983 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.270960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58532a1e-d1d4-43e3-b156-349d7950668c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.271216 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.271197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58532a1e-d1d4-43e3-b156-349d7950668c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.276829 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.276807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9d4\" (UniqueName: \"kubernetes.io/projected/58532a1e-d1d4-43e3-b156-349d7950668c-kube-api-access-jn9d4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks\" (UID: \"58532a1e-d1d4-43e3-b156-349d7950668c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.322427 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.322371    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:30.595447 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.595400    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"]
Sep 10 21:30:30.598226 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:30:30.598191    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58532a1e_d1d4_43e3_b156_349d7950668c.slice/crio-ce62db7146ef08475502d7b8e47ecea6463eab9c9cc87119ee584b7bec0e03c4 WatchSource:0}: Error finding container ce62db7146ef08475502d7b8e47ecea6463eab9c9cc87119ee584b7bec0e03c4: Status 404 returned error can't find the container with id ce62db7146ef08475502d7b8e47ecea6463eab9c9cc87119ee584b7bec0e03c4
Sep 10 21:30:30.768117 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:30.768079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks" event={"ID":"58532a1e-d1d4-43e3-b156-349d7950668c","Type":"ContainerStarted","Data":"ce62db7146ef08475502d7b8e47ecea6463eab9c9cc87119ee584b7bec0e03c4"}
Sep 10 21:30:31.774440 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:31.774388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks" event={"ID":"58532a1e-d1d4-43e3-b156-349d7950668c","Type":"ContainerStarted","Data":"4298e0c616e135cd04f717c05a8fd22f9051866f8f40f4318fbf52be6f001d40"}
Sep 10 21:30:31.795050 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:31.795001    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks" podStartSLOduration=2.5683817319999998 podStartE2EDuration="2.794983314s" totalImagesPullingTime="226.601582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:30:31.793065239 +0000 UTC m=+670.440979408" watchObservedRunningTime="2026-09-10 21:30:31.794983314 +0000 UTC m=+670.442897480"
Sep 10 21:30:32.213572 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:32.213533    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:30:33.934913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:33.934881    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:33.935279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:33.934962    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:33.947902 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:33.947877    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:34.799042 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:34.799015    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-klq8j"
Sep 10 21:30:40.323317 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:40.323268    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:40.323725 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:40.323348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:40.335428 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:40.335387    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:30:40.825105 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:30:40.825074    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks"
Sep 10 21:31:58.835256 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:31:58.835221    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-800802"]
Sep 10 21:31:58.838523 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:31:58.838508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-800802"
Sep 10 21:31:58.840698 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:31:58.840677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zd54x\""
Sep 10 21:31:58.844837 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:31:58.844816    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-800802"]
Sep 10 21:31:58.935890 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:31:58.935864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k87xk\" (UniqueName: \"kubernetes.io/projected/574744d4-3f47-440f-8456-41fe8f020f86-kube-api-access-k87xk\") pod \"test-curl-7564-800802\" (UID: \"574744d4-3f47-440f-8456-41fe8f020f86\") " pod="opendatahub/test-curl-7564-800802"
Sep 10 21:31:59.036911 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:31:59.036882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k87xk\" (UniqueName: \"kubernetes.io/projected/574744d4-3f47-440f-8456-41fe8f020f86-kube-api-access-k87xk\") pod \"test-curl-7564-800802\" (UID: \"574744d4-3f47-440f-8456-41fe8f020f86\") " pod="opendatahub/test-curl-7564-800802"
Sep 10 21:31:59.046114 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:31:59.046082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k87xk\" (UniqueName: \"kubernetes.io/projected/574744d4-3f47-440f-8456-41fe8f020f86-kube-api-access-k87xk\") pod \"test-curl-7564-800802\" (UID: \"574744d4-3f47-440f-8456-41fe8f020f86\") " pod="opendatahub/test-curl-7564-800802"
Sep 10 21:31:59.148326 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:31:59.148266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-800802"
Sep 10 21:31:59.414101 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:31:59.414078    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-800802"]
Sep 10 21:31:59.416529 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:31:59.416497    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod574744d4_3f47_440f_8456_41fe8f020f86.slice/crio-ff334a98419ec2171c2348272ccf0a25caed0e7d3e274c233649ede38bf7c4e9 WatchSource:0}: Error finding container ff334a98419ec2171c2348272ccf0a25caed0e7d3e274c233649ede38bf7c4e9: Status 404 returned error can't find the container with id ff334a98419ec2171c2348272ccf0a25caed0e7d3e274c233649ede38bf7c4e9
Sep 10 21:32:00.161051 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:00.161012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-800802" event={"ID":"574744d4-3f47-440f-8456-41fe8f020f86","Type":"ContainerStarted","Data":"ff334a98419ec2171c2348272ccf0a25caed0e7d3e274c233649ede38bf7c4e9"}
Sep 10 21:32:00.717682 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:00.717648    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:32:02.175472 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:02.175434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-800802" event={"ID":"574744d4-3f47-440f-8456-41fe8f020f86","Type":"ContainerStarted","Data":"5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca"}
Sep 10 21:32:02.189966 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:02.189914    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7564-800802" podStartSLOduration=2.010885016 podStartE2EDuration="4.189899362s" totalImagesPullingTime="2.179014346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:02.188484268 +0000 UTC m=+760.836398455" watchObservedRunningTime="2026-09-10 21:32:02.189899362 +0000 UTC m=+760.837813529"
Sep 10 21:32:02.767614 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:02.767577    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-800802"]
Sep 10 21:32:02.771223 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:02.771192    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-800802"]
Sep 10 21:32:02.892307 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:02.892227    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-e9957e"]
Sep 10 21:32:02.892640 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:02.892625    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="574744d4-3f47-440f-8456-41fe8f020f86" containerName="test-curl-7564-800802"
Sep 10 21:32:02.892686 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:02.892642    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="574744d4-3f47-440f-8456-41fe8f020f86" containerName="test-curl-7564-800802"
Sep 10 21:32:02.892720 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:02.892705    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="574744d4-3f47-440f-8456-41fe8f020f86" containerName="test-curl-7564-800802"
Sep 10 21:32:02.895825 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:02.895807    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-e9957e"
Sep 10 21:32:02.903491 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:02.903468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-e9957e"]
Sep 10 21:32:02.972895 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:02.972864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ghz9\" (UniqueName: \"kubernetes.io/projected/74cde713-fbe1-4b28-8802-7cc60a934cd9-kube-api-access-5ghz9\") pod \"test-curl-7564-e9957e\" (UID: \"74cde713-fbe1-4b28-8802-7cc60a934cd9\") " pod="opendatahub/test-curl-7564-e9957e"
Sep 10 21:32:03.074329 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:03.074296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghz9\" (UniqueName: \"kubernetes.io/projected/74cde713-fbe1-4b28-8802-7cc60a934cd9-kube-api-access-5ghz9\") pod \"test-curl-7564-e9957e\" (UID: \"74cde713-fbe1-4b28-8802-7cc60a934cd9\") " pod="opendatahub/test-curl-7564-e9957e"
Sep 10 21:32:03.090385 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:03.090353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghz9\" (UniqueName: \"kubernetes.io/projected/74cde713-fbe1-4b28-8802-7cc60a934cd9-kube-api-access-5ghz9\") pod \"test-curl-7564-e9957e\" (UID: \"74cde713-fbe1-4b28-8802-7cc60a934cd9\") " pod="opendatahub/test-curl-7564-e9957e"
Sep 10 21:32:03.182259 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:32:03.182184    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7564-800802" container="test-curl-7564-800802"
Sep 10 21:32:03.205272 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:03.205248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-e9957e"
Sep 10 21:32:03.470997 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:03.470960    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-e9957e"]
Sep 10 21:32:03.473732 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:32:03.473704    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74cde713_fbe1_4b28_8802_7cc60a934cd9.slice/crio-bcc575464e566208d75f7a4c6bcca66a662d435f2dbffcfec096c995774e173d WatchSource:0}: Error finding container bcc575464e566208d75f7a4c6bcca66a662d435f2dbffcfec096c995774e173d: Status 404 returned error can't find the container with id bcc575464e566208d75f7a4c6bcca66a662d435f2dbffcfec096c995774e173d
Sep 10 21:32:04.187855 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:04.187814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-e9957e" event={"ID":"74cde713-fbe1-4b28-8802-7cc60a934cd9","Type":"ContainerStarted","Data":"e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5"}
Sep 10 21:32:04.188470 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:04.187867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-e9957e" event={"ID":"74cde713-fbe1-4b28-8802-7cc60a934cd9","Type":"ContainerStarted","Data":"bcc575464e566208d75f7a4c6bcca66a662d435f2dbffcfec096c995774e173d"}
Sep 10 21:32:04.188470 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:04.187861    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7564-800802" podUID="574744d4-3f47-440f-8456-41fe8f020f86" containerName="test-curl-7564-800802" containerID="cri-o://5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca" gracePeriod=2
Sep 10 21:32:04.203336 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:04.203294    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7564-e9957e" podStartSLOduration=2.203279767 podStartE2EDuration="2.203279767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:04.201397573 +0000 UTC m=+762.849311740" watchObservedRunningTime="2026-09-10 21:32:04.203279767 +0000 UTC m=+762.851193947"
Sep 10 21:32:04.786612 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:04.786564    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-e9957e"]
Sep 10 21:32:04.789324 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:04.789261    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-e9957e"]
Sep 10 21:32:05.192740 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:32:05.192707    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7564-e9957e" container="test-curl-7564-e9957e"
Sep 10 21:32:05.594090 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.594000    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"]
Sep 10 21:32:05.594453 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.594436    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74cde713-fbe1-4b28-8802-7cc60a934cd9" containerName="test-curl-7564-e9957e"
Sep 10 21:32:05.594544 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.594456    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="74cde713-fbe1-4b28-8802-7cc60a934cd9" containerName="test-curl-7564-e9957e"
Sep 10 21:32:05.594596 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.594559    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="74cde713-fbe1-4b28-8802-7cc60a934cd9" containerName="test-curl-7564-e9957e"
Sep 10 21:32:05.597624 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.597604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.600073 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.600051    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-4b08e7-openshift-default-dockercfg-rtthc\""
Sep 10 21:32:05.600185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.600056    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:32:05.609321 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.609298    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"]
Sep 10 21:32:05.699994 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.699959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-credential-socket\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.700185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.699995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-istio-token\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.700185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.700041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994645f5-5899-4c60-ab5e-1443c2262811-istiod-ca-cert\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.700185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.700071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-envoy\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.700185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.700091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-certs\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.700185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.700157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-data\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.700381 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.700194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8zb\" (UniqueName: \"kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-kube-api-access-8j8zb\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.700381 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.700244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-socket\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.700381 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.700278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994645f5-5899-4c60-ab5e-1443c2262811-istio-podinfo\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.801369 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.801325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-socket\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.801369 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.801372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994645f5-5899-4c60-ab5e-1443c2262811-istio-podinfo\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.801631 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.801541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-credential-socket\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.801778 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.801737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-istio-token\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.801865 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.801810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-socket\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.802031 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.801965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994645f5-5899-4c60-ab5e-1443c2262811-istiod-ca-cert\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.802304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.802039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-credential-socket\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.802304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.802105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-envoy\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.802304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.802151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-certs\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.802304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.802179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-data\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.802304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.802274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8zb\" (UniqueName: \"kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-kube-api-access-8j8zb\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.802746 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.802727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994645f5-5899-4c60-ab5e-1443c2262811-istiod-ca-cert\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.802826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.802751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-certs\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.802862 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.802784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-data\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.804193 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.804167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994645f5-5899-4c60-ab5e-1443c2262811-istio-podinfo\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.804284 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.804273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-envoy\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.813987 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.813962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8zb\" (UniqueName: \"kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-kube-api-access-8j8zb\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.814126 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.814112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-istio-token\") pod \"e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") " pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:05.908951 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:05.908856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:06.186043 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:06.186020    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"]
Sep 10 21:32:06.188037 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:32:06.188013    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994645f5_5899_4c60_ab5e_1443c2262811.slice/crio-b03b0dd5c0973de79dd24bbbcdf5bbc99605765ba91e56435828736cae770380 WatchSource:0}: Error finding container b03b0dd5c0973de79dd24bbbcdf5bbc99605765ba91e56435828736cae770380: Status 404 returned error can't find the container with id b03b0dd5c0973de79dd24bbbcdf5bbc99605765ba91e56435828736cae770380
Sep 10 21:32:06.197824 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:06.197797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" event={"ID":"994645f5-5899-4c60-ab5e-1443c2262811","Type":"ContainerStarted","Data":"b03b0dd5c0973de79dd24bbbcdf5bbc99605765ba91e56435828736cae770380"}
Sep 10 21:32:06.198075 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:06.197904    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7564-e9957e" podUID="74cde713-fbe1-4b28-8802-7cc60a934cd9" containerName="test-curl-7564-e9957e" containerID="cri-o://e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5" gracePeriod=2
Sep 10 21:32:06.337994 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:06.337971    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-800802"
Sep 10 21:32:06.407975 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:06.407944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k87xk\" (UniqueName: \"kubernetes.io/projected/574744d4-3f47-440f-8456-41fe8f020f86-kube-api-access-k87xk\") pod \"574744d4-3f47-440f-8456-41fe8f020f86\" (UID: \"574744d4-3f47-440f-8456-41fe8f020f86\") "
Sep 10 21:32:06.410156 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:06.410135    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574744d4-3f47-440f-8456-41fe8f020f86-kube-api-access-k87xk" (OuterVolumeSpecName: "kube-api-access-k87xk") pod "574744d4-3f47-440f-8456-41fe8f020f86" (UID: "574744d4-3f47-440f-8456-41fe8f020f86"). InnerVolumeSpecName "kube-api-access-k87xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:32:06.509568 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:06.509544    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k87xk\" (UniqueName: \"kubernetes.io/projected/574744d4-3f47-440f-8456-41fe8f020f86-kube-api-access-k87xk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:32:07.205336 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:07.205293    2580 generic.go:356] "Generic (PLEG): container finished" podID="574744d4-3f47-440f-8456-41fe8f020f86" containerID="5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca" exitCode=137
Sep 10 21:32:07.205918 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:07.205349    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-800802"
Sep 10 21:32:07.205918 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:07.205369    2580 scope.go:117] "RemoveContainer" containerID="5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca"
Sep 10 21:32:07.216002 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:07.215985    2580 scope.go:117] "RemoveContainer" containerID="5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca"
Sep 10 21:32:07.216318 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:32:07.216298    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca\": container with ID starting with 5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca not found: ID does not exist" containerID="5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca"
Sep 10 21:32:07.216468 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:07.216326    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca"} err="failed to get container status \"5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca\": rpc error: code = NotFound desc = could not find container \"5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca\": container with ID starting with 5aae7f7953ab2cced8c8c07d43d91626c44eb6377270c531a6ce2e1e7e8dacca not found: ID does not exist"
Sep 10 21:32:07.217100 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:32:07.217074    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-800802\" is forbidden: User \"system:node:ip-10-0-135-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-226.ec2.internal' and this object" podUID="574744d4-3f47-440f-8456-41fe8f020f86" pod="opendatahub/test-curl-7564-800802"
Sep 10 21:32:07.821626 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:07.821595    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="574744d4-3f47-440f-8456-41fe8f020f86" path="/var/lib/kubelet/pods/574744d4-3f47-440f-8456-41fe8f020f86/volumes"
Sep 10 21:32:08.868905 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:08.868865    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:32:08.869179 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:08.868944    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:32:08.869179 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:08.868980    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:32:08.944258 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:08.944226    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-e9957e"
Sep 10 21:32:09.035285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.035255    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ghz9\" (UniqueName: \"kubernetes.io/projected/74cde713-fbe1-4b28-8802-7cc60a934cd9-kube-api-access-5ghz9\") pod \"74cde713-fbe1-4b28-8802-7cc60a934cd9\" (UID: \"74cde713-fbe1-4b28-8802-7cc60a934cd9\") "
Sep 10 21:32:09.037542 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.037502    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cde713-fbe1-4b28-8802-7cc60a934cd9-kube-api-access-5ghz9" (OuterVolumeSpecName: "kube-api-access-5ghz9") pod "74cde713-fbe1-4b28-8802-7cc60a934cd9" (UID: "74cde713-fbe1-4b28-8802-7cc60a934cd9"). InnerVolumeSpecName "kube-api-access-5ghz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:32:09.136831 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.136753    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ghz9\" (UniqueName: \"kubernetes.io/projected/74cde713-fbe1-4b28-8802-7cc60a934cd9-kube-api-access-5ghz9\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:32:09.222622 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.222582    2580 generic.go:356] "Generic (PLEG): container finished" podID="74cde713-fbe1-4b28-8802-7cc60a934cd9" containerID="e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5" exitCode=137
Sep 10 21:32:09.222622 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.222621    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-e9957e"
Sep 10 21:32:09.222846 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.222653    2580 scope.go:117] "RemoveContainer" containerID="e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5"
Sep 10 21:32:09.228264 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.228223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" event={"ID":"994645f5-5899-4c60-ab5e-1443c2262811","Type":"ContainerStarted","Data":"cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7"}
Sep 10 21:32:09.235590 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.235571    2580 scope.go:117] "RemoveContainer" containerID="e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5"
Sep 10 21:32:09.235915 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:32:09.235888    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5\": container with ID starting with e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5 not found: ID does not exist" containerID="e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5"
Sep 10 21:32:09.236045 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.235923    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5"} err="failed to get container status \"e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5\": rpc error: code = NotFound desc = could not find container \"e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5\": container with ID starting with e7c2203ccc8d0acbddd7e48026b30958fd2a4ce8a42264668401eba4bad8a7d5 not found: ID does not exist"
Sep 10 21:32:09.254352 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.254308    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" podStartSLOduration=1.576017257 podStartE2EDuration="4.254294154s" totalImagesPullingTime="2.678276897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:09.252569505 +0000 UTC m=+767.900483672" watchObservedRunningTime="2026-09-10 21:32:09.254294154 +0000 UTC m=+767.902208361"
Sep 10 21:32:09.254651 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:32:09.254627    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-e9957e\" is forbidden: User \"system:node:ip-10-0-135-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-226.ec2.internal' and this object" podUID="74cde713-fbe1-4b28-8802-7cc60a934cd9" pod="opendatahub/test-curl-7564-e9957e"
Sep 10 21:32:09.819659 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.819629    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74cde713-fbe1-4b28-8802-7cc60a934cd9" path="/var/lib/kubelet/pods/74cde713-fbe1-4b28-8802-7cc60a934cd9/volumes"
Sep 10 21:32:09.909761 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.909726    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:09.914595 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:09.914568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:10.233271 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:10.233246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:10.234315 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:10.234295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:12.310833 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.310803    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"]
Sep 10 21:32:12.315546 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.315530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:12.317986 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.317961    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-4b08e7-serving-cert\""
Sep 10 21:32:12.318669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.318650    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:32:12.318780 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.318679    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 21:32:12.318780 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.318679    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nmksg\""
Sep 10 21:32:12.318780 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.318650    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 21:32:12.323272 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.323003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"]
Sep 10 21:32:12.467022 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.466985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204e9152-d0bf-4a1d-988d-ad86c0f303f2-openshift-service-ca\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:12.467191 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.467031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-metrics-certs\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:12.467191 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.467052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9khfv\" (UniqueName: \"kubernetes.io/projected/204e9152-d0bf-4a1d-988d-ad86c0f303f2-kube-api-access-9khfv\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:12.467191 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.467166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-maas-api-tls\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:12.568401 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.568319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204e9152-d0bf-4a1d-988d-ad86c0f303f2-openshift-service-ca\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:12.568401 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.568362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-metrics-certs\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:12.568401 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.568379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9khfv\" (UniqueName: \"kubernetes.io/projected/204e9152-d0bf-4a1d-988d-ad86c0f303f2-kube-api-access-9khfv\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:12.568636 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.568461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-maas-api-tls\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:12.568636 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:32:12.568559    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-serving-cert: secret "maas-api-e2e-autores-4b08e7-serving-cert" not found
Sep 10 21:32:12.568636 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:32:12.568625    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-maas-api-tls podName:204e9152-d0bf-4a1d-988d-ad86c0f303f2 nodeName:}" failed. No retries permitted until 2026-09-10 21:32:13.068607546 +0000 UTC m=+771.716521714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-maas-api-tls") pod "maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx" (UID: "204e9152-d0bf-4a1d-988d-ad86c0f303f2") : secret "maas-api-e2e-autores-4b08e7-serving-cert" not found
Sep 10 21:32:12.569022 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.569004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204e9152-d0bf-4a1d-988d-ad86c0f303f2-openshift-service-ca\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:12.570929 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.570909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-metrics-certs\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:12.579335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:12.579308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9khfv\" (UniqueName: \"kubernetes.io/projected/204e9152-d0bf-4a1d-988d-ad86c0f303f2-kube-api-access-9khfv\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:13.026742 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.026707    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"]
Sep 10 21:32:13.030190 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.030173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:32:13.036998 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.036971    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-4b08e7-dockercfg-2nxs6\""
Sep 10 21:32:13.037368 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.037352    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-4b08e7\""
Sep 10 21:32:13.046698 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.046672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"]
Sep 10 21:32:13.074847 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.074815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-maas-api-tls\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:13.077331 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.077308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-maas-api-tls\") pod \"maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:13.175373 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.175335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chgjk\" (UniqueName: \"kubernetes.io/projected/ed4bf264-50b2-4161-b75f-1a577782dc30-kube-api-access-chgjk\") pod \"payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt\" (UID: \"ed4bf264-50b2-4161-b75f-1a577782dc30\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:32:13.175562 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.175438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4bf264-50b2-4161-b75f-1a577782dc30-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt\" (UID: \"ed4bf264-50b2-4161-b75f-1a577782dc30\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:32:13.226814 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.226780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:13.276573 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.276279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgjk\" (UniqueName: \"kubernetes.io/projected/ed4bf264-50b2-4161-b75f-1a577782dc30-kube-api-access-chgjk\") pod \"payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt\" (UID: \"ed4bf264-50b2-4161-b75f-1a577782dc30\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:32:13.276573 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.276440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4bf264-50b2-4161-b75f-1a577782dc30-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt\" (UID: \"ed4bf264-50b2-4161-b75f-1a577782dc30\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:32:13.277370 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.277319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4bf264-50b2-4161-b75f-1a577782dc30-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt\" (UID: \"ed4bf264-50b2-4161-b75f-1a577782dc30\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:32:13.292190 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.292163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chgjk\" (UniqueName: \"kubernetes.io/projected/ed4bf264-50b2-4161-b75f-1a577782dc30-kube-api-access-chgjk\") pod \"payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt\" (UID: \"ed4bf264-50b2-4161-b75f-1a577782dc30\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:32:13.340203 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.340167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:32:13.498194 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.498152    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"]
Sep 10 21:32:13.500213 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:32:13.500186    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204e9152_d0bf_4a1d_988d_ad86c0f303f2.slice/crio-0828ece610e37d95b278f967b50084b55fd43b4e33e7c544735c610c2152763e WatchSource:0}: Error finding container 0828ece610e37d95b278f967b50084b55fd43b4e33e7c544735c610c2152763e: Status 404 returned error can't find the container with id 0828ece610e37d95b278f967b50084b55fd43b4e33e7c544735c610c2152763e
Sep 10 21:32:13.614090 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:13.614020    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"]
Sep 10 21:32:13.617202 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:32:13.617173    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4bf264_50b2_4161_b75f_1a577782dc30.slice/crio-b9f6e6a8c2be76e70f5e5e6cf8f81be7320c22b90adc49cc811eba3a1f9396d4 WatchSource:0}: Error finding container b9f6e6a8c2be76e70f5e5e6cf8f81be7320c22b90adc49cc811eba3a1f9396d4: Status 404 returned error can't find the container with id b9f6e6a8c2be76e70f5e5e6cf8f81be7320c22b90adc49cc811eba3a1f9396d4
Sep 10 21:32:14.255820 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:14.255773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt" event={"ID":"ed4bf264-50b2-4161-b75f-1a577782dc30","Type":"ContainerStarted","Data":"b9f6e6a8c2be76e70f5e5e6cf8f81be7320c22b90adc49cc811eba3a1f9396d4"}
Sep 10 21:32:14.257828 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:14.257762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx" event={"ID":"204e9152-d0bf-4a1d-988d-ad86c0f303f2","Type":"ContainerStarted","Data":"0828ece610e37d95b278f967b50084b55fd43b4e33e7c544735c610c2152763e"}
Sep 10 21:32:16.271526 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:16.271487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt" event={"ID":"ed4bf264-50b2-4161-b75f-1a577782dc30","Type":"ContainerStarted","Data":"9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927"}
Sep 10 21:32:16.271924 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:16.271603    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:32:16.273273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:16.273244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx" event={"ID":"204e9152-d0bf-4a1d-988d-ad86c0f303f2","Type":"ContainerStarted","Data":"5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b"}
Sep 10 21:32:16.273393 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:16.273377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:16.291851 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:16.291806    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt" podStartSLOduration=0.750851551 podStartE2EDuration="3.291792988s" totalImagesPullingTime="2.540941437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:16.289628978 +0000 UTC m=+774.937543148" watchObservedRunningTime="2026-09-10 21:32:16.291792988 +0000 UTC m=+774.939707196"
Sep 10 21:32:16.308499 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:16.308460    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx" podStartSLOduration=1.652780672 podStartE2EDuration="4.308446181s" totalImagesPullingTime="2.655665509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:16.307328927 +0000 UTC m=+774.955243116" watchObservedRunningTime="2026-09-10 21:32:16.308446181 +0000 UTC m=+774.956360344"
Sep 10 21:32:16.508626 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:16.508589    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:32:22.283676 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:22.283646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:32:22.900985 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:22.900952    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"]
Sep 10 21:32:22.905735 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:22.905719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:22.908202 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:22.908184    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-15ae234f-kserve-self-signed-certs\""
Sep 10 21:32:22.914164 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:22.914142    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"]
Sep 10 21:32:22.978107 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:22.978061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tls-certs\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:22.978269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:22.978117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-dshm\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:22.978269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:22.978165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-model-cache\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:22.978269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:22.978246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-home\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:22.978269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:22.978265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tmp-dir\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:22.978527 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:22.978297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftb6\" (UniqueName: \"kubernetes.io/projected/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-kube-api-access-tftb6\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.079915 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.079876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tls-certs\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.080152 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.080130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-dshm\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.080285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.080272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-model-cache\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.080502 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.080463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-home\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.081001 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.080982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tmp-dir\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.081501 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.081480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftb6\" (UniqueName: \"kubernetes.io/projected/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-kube-api-access-tftb6\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.082215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.082194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-model-cache\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.082599 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.082581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-home\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.082967 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.082947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tmp-dir\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.085039 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.085020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-dshm\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.087650 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.087625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tls-certs\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.090960 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.090938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftb6\" (UniqueName: \"kubernetes.io/projected/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-kube-api-access-tftb6\") pod \"e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") " pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.215457 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.215390    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:23.499900 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:23.499877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"]
Sep 10 21:32:23.501711 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:32:23.501678    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b35199a_3ac9_4efa_bf8e_50b735b6ce7d.slice/crio-8a51114c48f9c1591e235669a0ab9b1617b808e05d8c803bd3bac149cb6828aa WatchSource:0}: Error finding container 8a51114c48f9c1591e235669a0ab9b1617b808e05d8c803bd3bac149cb6828aa: Status 404 returned error can't find the container with id 8a51114c48f9c1591e235669a0ab9b1617b808e05d8c803bd3bac149cb6828aa
Sep 10 21:32:24.316036 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:24.315999    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz" event={"ID":"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d","Type":"ContainerStarted","Data":"1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d"}
Sep 10 21:32:24.316210 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:24.316050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz" event={"ID":"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d","Type":"ContainerStarted","Data":"8a51114c48f9c1591e235669a0ab9b1617b808e05d8c803bd3bac149cb6828aa"}
Sep 10 21:32:24.335736 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:24.335687    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz" podStartSLOduration=2.3356666170000002 podStartE2EDuration="2.335666617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:24.332910614 +0000 UTC m=+782.980824793" watchObservedRunningTime="2026-09-10 21:32:24.335666617 +0000 UTC m=+782.983580786"
Sep 10 21:32:26.217592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:26.216761    2580 patch_prober.go:28] interesting pod/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": read tcp 10.133.0.2:37074->10.133.0.34:15021: read: connection reset by peer" start-of-body=
Sep 10 21:32:26.217592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:26.216825    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": read tcp 10.133.0.2:37074->10.133.0.34:15021: read: connection reset by peer"
Sep 10 21:32:26.327939 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:26.327902    2580 generic.go:356] "Generic (PLEG): container finished" podID="994645f5-5899-4c60-ab5e-1443c2262811" containerID="cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7" exitCode=0
Sep 10 21:32:26.328091 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:26.327978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" event={"ID":"994645f5-5899-4c60-ab5e-1443c2262811","Type":"ContainerDied","Data":"cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7"}
Sep 10 21:32:26.328480 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:26.328461    2580 scope.go:117] "RemoveContainer" containerID="cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7"
Sep 10 21:32:26.329480 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:26.329455    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:32:26.329545 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:26.329521    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:32:26.329581 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:26.329551    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:32:27.280109 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:27.280077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:32:27.335292 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:27.335257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" event={"ID":"994645f5-5899-4c60-ab5e-1443c2262811","Type":"ContainerStarted","Data":"d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133"}
Sep 10 21:32:27.909959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:27.909920    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:27.910155 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:27.910132    2580 patch_prober.go:28] interesting pod/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 10 21:32:27.910198 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:27.910184    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 10 21:32:28.909717 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:28.909683    2580 patch_prober.go:28] interesting pod/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 10 21:32:28.910135 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:28.909738    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 10 21:32:29.909872 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:29.909835    2580 patch_prober.go:28] interesting pod/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 10 21:32:29.910243 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:29.909896    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 10 21:32:29.929949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:29.929922    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"]
Sep 10 21:32:29.933674 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:29.933652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:29.935710 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:29.935691    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5bba43-serving-cert\""
Sep 10 21:32:29.942723 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:29.942698    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"]
Sep 10 21:32:30.043613 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.043583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-maas-api-tls\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.043613 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.043618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-metrics-certs\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.043831 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.043683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwv9f\" (UniqueName: \"kubernetes.io/projected/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-kube-api-access-bwv9f\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.043831 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.043785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.144806 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.144768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.144995 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.144817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-maas-api-tls\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.144995 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.144835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-metrics-certs\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.144995 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.144868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwv9f\" (UniqueName: \"kubernetes.io/projected/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-kube-api-access-bwv9f\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.145450 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.145429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.147353 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.147331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-metrics-certs\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.147472 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.147354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-maas-api-tls\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.153235 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.153211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwv9f\" (UniqueName: \"kubernetes.io/projected/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-kube-api-access-bwv9f\") pod \"maas-api-e2e-inf-b-5bba43-78c7665787-nzkct\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.245328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.245291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:30.548694 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.548669    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"]
Sep 10 21:32:30.551016 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:32:30.550991    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f6f6db_e60b_42c1_a6c4_43c7cc310bc8.slice/crio-72ab462e65dafcef46bc6b27c30e83c53589ef9c7f7bf52ba2de2390803d921f WatchSource:0}: Error finding container 72ab462e65dafcef46bc6b27c30e83c53589ef9c7f7bf52ba2de2390803d921f: Status 404 returned error can't find the container with id 72ab462e65dafcef46bc6b27c30e83c53589ef9c7f7bf52ba2de2390803d921f
Sep 10 21:32:30.573626 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.573583    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:32:30.768437 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.767993    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"]
Sep 10 21:32:30.774772 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.774741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:32:30.775644 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.775616    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"]
Sep 10 21:32:30.777939 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.777912    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5bba43-dockercfg-8wljm\""
Sep 10 21:32:30.778213 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.778190    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5bba43\""
Sep 10 21:32:30.785462 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.785435    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"]
Sep 10 21:32:30.785573 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.785526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:32:30.793458 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.793238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"]
Sep 10 21:32:30.850767 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.850733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzhwh\" (UniqueName: \"kubernetes.io/projected/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-kube-api-access-rzhwh\") pod \"payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48\" (UID: \"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:32:30.850927 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.850774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00297454-d8b5-422c-82b0-abdfbad07950-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl\" (UID: \"00297454-d8b5-422c-82b0-abdfbad07950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:32:30.850927 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.850823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfqd\" (UniqueName: \"kubernetes.io/projected/00297454-d8b5-422c-82b0-abdfbad07950-kube-api-access-jjfqd\") pod \"payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl\" (UID: \"00297454-d8b5-422c-82b0-abdfbad07950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:32:30.850927 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.850910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48\" (UID: \"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:32:30.909328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.909298    2580 patch_prober.go:28] interesting pod/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 10 21:32:30.909445 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.909359    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 10 21:32:30.951950 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.951903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfqd\" (UniqueName: \"kubernetes.io/projected/00297454-d8b5-422c-82b0-abdfbad07950-kube-api-access-jjfqd\") pod \"payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl\" (UID: \"00297454-d8b5-422c-82b0-abdfbad07950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:32:30.952391 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.952001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48\" (UID: \"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:32:30.952391 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.952122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzhwh\" (UniqueName: \"kubernetes.io/projected/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-kube-api-access-rzhwh\") pod \"payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48\" (UID: \"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:32:30.952391 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.952161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00297454-d8b5-422c-82b0-abdfbad07950-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl\" (UID: \"00297454-d8b5-422c-82b0-abdfbad07950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:32:30.952795 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.952773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48\" (UID: \"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:32:30.953116 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.953093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00297454-d8b5-422c-82b0-abdfbad07950-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl\" (UID: \"00297454-d8b5-422c-82b0-abdfbad07950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:32:30.965479 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.963771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfqd\" (UniqueName: \"kubernetes.io/projected/00297454-d8b5-422c-82b0-abdfbad07950-kube-api-access-jjfqd\") pod \"payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl\" (UID: \"00297454-d8b5-422c-82b0-abdfbad07950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:32:30.965809 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:30.965752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzhwh\" (UniqueName: \"kubernetes.io/projected/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-kube-api-access-rzhwh\") pod \"payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48\" (UID: \"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:32:31.096653 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:31.096574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:32:31.102702 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:31.102677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:32:31.357807 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:31.357700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct" event={"ID":"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8","Type":"ContainerStarted","Data":"d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca"}
Sep 10 21:32:31.357972 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:31.357807    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct" event={"ID":"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8","Type":"ContainerStarted","Data":"72ab462e65dafcef46bc6b27c30e83c53589ef9c7f7bf52ba2de2390803d921f"}
Sep 10 21:32:31.358516 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:31.358484    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:31.380082 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:31.380018    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct" podStartSLOduration=2.033752122 podStartE2EDuration="2.37999537s" totalImagesPullingTime="346.243248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:31.378772285 +0000 UTC m=+790.026686453" watchObservedRunningTime="2026-09-10 21:32:31.37999537 +0000 UTC m=+790.027909540"
Sep 10 21:32:31.385521 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:31.385489    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"]
Sep 10 21:32:31.386943 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:32:31.386913    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7617ce_25db_48a7_b2c1_9e86f3cf9c40.slice/crio-a6c0f39133b2b5da84d5dcd999ad71a3179c112def895361ecfff7f633263bfb WatchSource:0}: Error finding container a6c0f39133b2b5da84d5dcd999ad71a3179c112def895361ecfff7f633263bfb: Status 404 returned error can't find the container with id a6c0f39133b2b5da84d5dcd999ad71a3179c112def895361ecfff7f633263bfb
Sep 10 21:32:31.406111 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:31.406087    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"]
Sep 10 21:32:31.409512 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:32:31.409487    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00297454_d8b5_422c_82b0_abdfbad07950.slice/crio-9dd839e7e64b885794d7b929c8357d991972e31e9b3dc153a96a17e2001a7b09 WatchSource:0}: Error finding container 9dd839e7e64b885794d7b929c8357d991972e31e9b3dc153a96a17e2001a7b09: Status 404 returned error can't find the container with id 9dd839e7e64b885794d7b929c8357d991972e31e9b3dc153a96a17e2001a7b09
Sep 10 21:32:31.909625 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:31.909573    2580 patch_prober.go:28] interesting pod/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 10 21:32:31.909837 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:31.909650    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 10 21:32:32.366552 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:32.366512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl" event={"ID":"00297454-d8b5-422c-82b0-abdfbad07950","Type":"ContainerStarted","Data":"969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534"}
Sep 10 21:32:32.367035 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:32.366579    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl" event={"ID":"00297454-d8b5-422c-82b0-abdfbad07950","Type":"ContainerStarted","Data":"9dd839e7e64b885794d7b929c8357d991972e31e9b3dc153a96a17e2001a7b09"}
Sep 10 21:32:32.367035 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:32.366607    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:32:32.368622 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:32.368588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48" event={"ID":"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40","Type":"ContainerStarted","Data":"43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e"}
Sep 10 21:32:32.368777 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:32.368649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48" event={"ID":"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40","Type":"ContainerStarted","Data":"a6c0f39133b2b5da84d5dcd999ad71a3179c112def895361ecfff7f633263bfb"}
Sep 10 21:32:32.368777 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:32.368694    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:32:32.383347 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:32.383302    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl" podStartSLOduration=2.383288158 podStartE2EDuration="2.383288158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:32.381437898 +0000 UTC m=+791.029352063" watchObservedRunningTime="2026-09-10 21:32:32.383288158 +0000 UTC m=+791.031202366"
Sep 10 21:32:32.398624 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:32.398575    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48" podStartSLOduration=2.398556224 podStartE2EDuration="2.398556224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:32.396946109 +0000 UTC m=+791.044860277" watchObservedRunningTime="2026-09-10 21:32:32.398556224 +0000 UTC m=+791.046470394"
Sep 10 21:32:32.912775 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:32.912749    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:32.913041 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:32.913006    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:32.913694 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:32.913677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:32:33.216522 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:33.216489    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:33.216751 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:33.216735    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:33.228525 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:33.228501    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:33.385949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:33.385918    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:32:38.381074 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:38.381046    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:32:43.377095 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:43.377009    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:32:43.377095 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:43.377081    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:32:46.502919 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:32:46.502884    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:33:04.813760 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:04.813728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:33:29.421509 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:29.421475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:33:43.590341 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.590305    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"]
Sep 10 21:33:43.594926 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.594904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.597295 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.597274    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-15ae234f-kserve-self-signed-certs\""
Sep 10 21:33:43.603223 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.603201    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"]
Sep 10 21:33:43.648989 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.648962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk476\" (UniqueName: \"kubernetes.io/projected/38b31228-5382-414c-bbcc-b9b03101b3e9-kube-api-access-dk476\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.649131 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.649007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-model-cache\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.649131 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.649038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38b31228-5382-414c-bbcc-b9b03101b3e9-tls-certs\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.649131 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.649105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-dshm\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.649249 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.649130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-tmp-dir\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.649249 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.649166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-home\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.750612 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.750583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-home\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.750786 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.750693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk476\" (UniqueName: \"kubernetes.io/projected/38b31228-5382-414c-bbcc-b9b03101b3e9-kube-api-access-dk476\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.750786 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.750728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-model-cache\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.750786 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.750755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38b31228-5382-414c-bbcc-b9b03101b3e9-tls-certs\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.750949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.750809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-dshm\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.750949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.750834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-tmp-dir\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.751053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.750976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-home\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.751107 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.751060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-model-cache\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.751181 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.751161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-tmp-dir\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.753206 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.753179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-dshm\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.753336 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.753318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38b31228-5382-414c-bbcc-b9b03101b3e9-tls-certs\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.758856 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.758834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk476\" (UniqueName: \"kubernetes.io/projected/38b31228-5382-414c-bbcc-b9b03101b3e9-kube-api-access-dk476\") pod \"e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") " pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:43.905167 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:43.905091    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:44.176782 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:44.176755    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"]
Sep 10 21:33:44.179561 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:33:44.179533    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38b31228_5382_414c_bbcc_b9b03101b3e9.slice/crio-b940a0e15478881544642f0e37200231da414cfb40260727801706cdecd6f97e WatchSource:0}: Error finding container b940a0e15478881544642f0e37200231da414cfb40260727801706cdecd6f97e: Status 404 returned error can't find the container with id b940a0e15478881544642f0e37200231da414cfb40260727801706cdecd6f97e
Sep 10 21:33:44.730535 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:44.730496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9" event={"ID":"38b31228-5382-414c-bbcc-b9b03101b3e9","Type":"ContainerStarted","Data":"970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b"}
Sep 10 21:33:44.730535 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:44.730540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9" event={"ID":"38b31228-5382-414c-bbcc-b9b03101b3e9","Type":"ContainerStarted","Data":"b940a0e15478881544642f0e37200231da414cfb40260727801706cdecd6f97e"}
Sep 10 21:33:44.746261 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:44.746208    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9" podStartSLOduration=1.746191287 podStartE2EDuration="1.746191287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:33:44.745823891 +0000 UTC m=+863.393738057" watchObservedRunningTime="2026-09-10 21:33:44.746191287 +0000 UTC m=+863.394105455"
Sep 10 21:33:44.805404 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:44.805373    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:33:53.707442 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.707384    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"]
Sep 10 21:33:53.711035 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.711018    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.714145 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.714118    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4b08e7\"/\"kube-root-ca.crt\""
Sep 10 21:33:53.714267 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.714118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4b08e7\"/\"default-dockercfg-k8snd\""
Sep 10 21:33:53.714267 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.714164    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4b08e7\"/\"autores-model-4b08e7-kserve-self-signed-certs\""
Sep 10 21:33:53.714267 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.714163    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4b08e7\"/\"openshift-service-ca.crt\""
Sep 10 21:33:53.721819 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.721797    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"]
Sep 10 21:33:53.746027 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.746000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-tmp-dir\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.746146 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.746038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-dshm\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.746146 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.746061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2q4h\" (UniqueName: \"kubernetes.io/projected/19e74f05-efff-470b-9ab9-91495b7e6251-kube-api-access-w2q4h\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.746146 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.746086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-model-cache\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.746253 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.746156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-home\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.746253 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.746187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19e74f05-efff-470b-9ab9-91495b7e6251-tls-certs\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.846858 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.846819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-home\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.846858 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.846862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19e74f05-efff-470b-9ab9-91495b7e6251-tls-certs\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.847102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.846911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-tmp-dir\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.847102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.846933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-dshm\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.847102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.846949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2q4h\" (UniqueName: \"kubernetes.io/projected/19e74f05-efff-470b-9ab9-91495b7e6251-kube-api-access-w2q4h\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.847102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.846966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-model-cache\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.847305 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.847249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-home\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.847305 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.847264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-tmp-dir\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.847393 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.847353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-model-cache\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.849237 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.849217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-dshm\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.849528 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.849511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19e74f05-efff-470b-9ab9-91495b7e6251-tls-certs\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.855264 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.855241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2q4h\" (UniqueName: \"kubernetes.io/projected/19e74f05-efff-470b-9ab9-91495b7e6251-kube-api-access-w2q4h\") pod \"autores-model-4b08e7-kserve-54b7c576cf-b5pjq\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") " pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:53.905217 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.905183    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:53.905389 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.905254    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:53.919182 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:53.919159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:54.020791 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:54.020633    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:33:54.290852 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:54.290827    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"]
Sep 10 21:33:54.292760 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:33:54.292730    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19e74f05_efff_470b_9ab9_91495b7e6251.slice/crio-c2bbf9ce97db978a60f1749620ad6c8e2986bf3ba1a2aea5e5ee597093b02234 WatchSource:0}: Error finding container c2bbf9ce97db978a60f1749620ad6c8e2986bf3ba1a2aea5e5ee597093b02234: Status 404 returned error can't find the container with id c2bbf9ce97db978a60f1749620ad6c8e2986bf3ba1a2aea5e5ee597093b02234
Sep 10 21:33:54.783118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:54.783081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq" event={"ID":"19e74f05-efff-470b-9ab9-91495b7e6251","Type":"ContainerStarted","Data":"2c464f06700d55e1154c70f3cc518c5ff340f54849632af443e0ac0de5444b41"}
Sep 10 21:33:54.783118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:54.783126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq" event={"ID":"19e74f05-efff-470b-9ab9-91495b7e6251","Type":"ContainerStarted","Data":"c2bbf9ce97db978a60f1749620ad6c8e2986bf3ba1a2aea5e5ee597093b02234"}
Sep 10 21:33:54.797400 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:54.797367    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:33:54.801197 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:33:54.801157    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq" podStartSLOduration=1.801140924 podStartE2EDuration="1.801140924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:33:54.800205278 +0000 UTC m=+873.448119445" watchObservedRunningTime="2026-09-10 21:33:54.801140924 +0000 UTC m=+873.449055091"
Sep 10 21:34:01.523304 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:01.523257    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:34:04.021398 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:04.021356    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:34:04.021883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:04.021446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:34:04.033711 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:04.033684    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:34:04.838923 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:04.838894    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:34:20.141049 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:20.140960    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"]
Sep 10 21:34:20.843088 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:20.843046    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"]
Sep 10 21:34:25.547598 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:25.547549    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:34:27.314335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:27.314304    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"]
Sep 10 21:34:31.997778 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:31.997744    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"]
Sep 10 21:34:31.998285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:31.998015    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx" podUID="204e9152-d0bf-4a1d-988d-ad86c0f303f2" containerName="maas-api" containerID="cri-o://5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b" gracePeriod=30
Sep 10 21:34:32.075716 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.075681    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"]
Sep 10 21:34:32.075908 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.075880    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt" podUID="ed4bf264-50b2-4161-b75f-1a577782dc30" containerName="payload-pre-processing" containerID="cri-o://9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927" gracePeriod=30
Sep 10 21:34:32.353929 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.353899    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:34:32.357013 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.356987    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:34:32.431854 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.431823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4bf264-50b2-4161-b75f-1a577782dc30-plugins-config-volume\") pod \"ed4bf264-50b2-4161-b75f-1a577782dc30\" (UID: \"ed4bf264-50b2-4161-b75f-1a577782dc30\") "
Sep 10 21:34:32.431854 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.431861    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chgjk\" (UniqueName: \"kubernetes.io/projected/ed4bf264-50b2-4161-b75f-1a577782dc30-kube-api-access-chgjk\") pod \"ed4bf264-50b2-4161-b75f-1a577782dc30\" (UID: \"ed4bf264-50b2-4161-b75f-1a577782dc30\") "
Sep 10 21:34:32.432201 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.431888    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9khfv\" (UniqueName: \"kubernetes.io/projected/204e9152-d0bf-4a1d-988d-ad86c0f303f2-kube-api-access-9khfv\") pod \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") "
Sep 10 21:34:32.432201 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.431916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204e9152-d0bf-4a1d-988d-ad86c0f303f2-openshift-service-ca\") pod \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") "
Sep 10 21:34:32.432201 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.431971    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-maas-api-tls\") pod \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") "
Sep 10 21:34:32.432201 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.432038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-metrics-certs\") pod \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\" (UID: \"204e9152-d0bf-4a1d-988d-ad86c0f303f2\") "
Sep 10 21:34:32.432435 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.432256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4bf264-50b2-4161-b75f-1a577782dc30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed4bf264-50b2-4161-b75f-1a577782dc30" (UID: "ed4bf264-50b2-4161-b75f-1a577782dc30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:34:32.432435 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.432275    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204e9152-d0bf-4a1d-988d-ad86c0f303f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "204e9152-d0bf-4a1d-988d-ad86c0f303f2" (UID: "204e9152-d0bf-4a1d-988d-ad86c0f303f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:34:32.432435 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.432384    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204e9152-d0bf-4a1d-988d-ad86c0f303f2-openshift-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:32.432435 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.432402    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4bf264-50b2-4161-b75f-1a577782dc30-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:32.434390 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.434356    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "204e9152-d0bf-4a1d-988d-ad86c0f303f2" (UID: "204e9152-d0bf-4a1d-988d-ad86c0f303f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:34:32.434521 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.434442    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "204e9152-d0bf-4a1d-988d-ad86c0f303f2" (UID: "204e9152-d0bf-4a1d-988d-ad86c0f303f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:34:32.434521 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.434486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204e9152-d0bf-4a1d-988d-ad86c0f303f2-kube-api-access-9khfv" (OuterVolumeSpecName: "kube-api-access-9khfv") pod "204e9152-d0bf-4a1d-988d-ad86c0f303f2" (UID: "204e9152-d0bf-4a1d-988d-ad86c0f303f2"). InnerVolumeSpecName "kube-api-access-9khfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:34:32.434594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.434533    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4bf264-50b2-4161-b75f-1a577782dc30-kube-api-access-chgjk" (OuterVolumeSpecName: "kube-api-access-chgjk") pod "ed4bf264-50b2-4161-b75f-1a577782dc30" (UID: "ed4bf264-50b2-4161-b75f-1a577782dc30"). InnerVolumeSpecName "kube-api-access-chgjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:34:32.533211 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.533128    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-metrics-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:32.533211 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.533155    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chgjk\" (UniqueName: \"kubernetes.io/projected/ed4bf264-50b2-4161-b75f-1a577782dc30-kube-api-access-chgjk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:32.533211 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.533165    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9khfv\" (UniqueName: \"kubernetes.io/projected/204e9152-d0bf-4a1d-988d-ad86c0f303f2-kube-api-access-9khfv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:32.533211 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.533174    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9152-d0bf-4a1d-988d-ad86c0f303f2-maas-api-tls\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:32.968555 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.968523    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed4bf264-50b2-4161-b75f-1a577782dc30" containerID="9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927" exitCode=0
Sep 10 21:34:32.968753 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.968567    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"
Sep 10 21:34:32.968753 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.968599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt" event={"ID":"ed4bf264-50b2-4161-b75f-1a577782dc30","Type":"ContainerDied","Data":"9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927"}
Sep 10 21:34:32.968753 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.968644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt" event={"ID":"ed4bf264-50b2-4161-b75f-1a577782dc30","Type":"ContainerDied","Data":"b9f6e6a8c2be76e70f5e5e6cf8f81be7320c22b90adc49cc811eba3a1f9396d4"}
Sep 10 21:34:32.968753 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.968673    2580 scope.go:117] "RemoveContainer" containerID="9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927"
Sep 10 21:34:32.970202 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.970159    2580 generic.go:356] "Generic (PLEG): container finished" podID="204e9152-d0bf-4a1d-988d-ad86c0f303f2" containerID="5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b" exitCode=0
Sep 10 21:34:32.970328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.970211    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"
Sep 10 21:34:32.970328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.970213    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx" event={"ID":"204e9152-d0bf-4a1d-988d-ad86c0f303f2","Type":"ContainerDied","Data":"5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b"}
Sep 10 21:34:32.970328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.970256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx" event={"ID":"204e9152-d0bf-4a1d-988d-ad86c0f303f2","Type":"ContainerDied","Data":"0828ece610e37d95b278f967b50084b55fd43b4e33e7c544735c610c2152763e"}
Sep 10 21:34:32.977921 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.977904    2580 scope.go:117] "RemoveContainer" containerID="9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927"
Sep 10 21:34:32.978208 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:34:32.978191    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927\": container with ID starting with 9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927 not found: ID does not exist" containerID="9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927"
Sep 10 21:34:32.978273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.978217    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927"} err="failed to get container status \"9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927\": rpc error: code = NotFound desc = could not find container \"9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927\": container with ID starting with 9b82695e345e32ec6d611d589b0e7feeb4da65d6dadbe27324ddd38733e49927 not found: ID does not exist"
Sep 10 21:34:32.978273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.978234    2580 scope.go:117] "RemoveContainer" containerID="5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b"
Sep 10 21:34:32.986274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.986258    2580 scope.go:117] "RemoveContainer" containerID="5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b"
Sep 10 21:34:32.986558 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:34:32.986540    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b\": container with ID starting with 5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b not found: ID does not exist" containerID="5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b"
Sep 10 21:34:32.986622 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:32.986568    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b"} err="failed to get container status \"5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b\": rpc error: code = NotFound desc = could not find container \"5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b\": container with ID starting with 5ecd460a7bdc8d0bf3a173ea82ee2bc57ca8e8b7aa52458f9afba84bae942c2b not found: ID does not exist"
Sep 10 21:34:33.015423 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:33.015372    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"]
Sep 10 21:34:33.022498 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:33.022472    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4b08e7-5db5689f9d-bnvjt"]
Sep 10 21:34:33.045154 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:33.045127    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"]
Sep 10 21:34:33.048785 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:33.048764    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx"]
Sep 10 21:34:33.820980 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:33.820943    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="204e9152-d0bf-4a1d-988d-ad86c0f303f2" path="/var/lib/kubelet/pods/204e9152-d0bf-4a1d-988d-ad86c0f303f2/volumes"
Sep 10 21:34:33.821437 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:33.821398    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed4bf264-50b2-4161-b75f-1a577782dc30" path="/var/lib/kubelet/pods/ed4bf264-50b2-4161-b75f-1a577782dc30/volumes"
Sep 10 21:34:35.162718 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.162682    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9" podUID="38b31228-5382-414c-bbcc-b9b03101b3e9" containerName="main" containerID="cri-o://970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b" gracePeriod=45
Sep 10 21:34:35.279141 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.279094    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b08e7-59696c6d67-h9bcx" podUID="204e9152-d0bf-4a1d-988d-ad86c0f303f2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.35:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 21:34:35.409913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.409884    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:34:35.462080 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-dshm\") pod \"38b31228-5382-414c-bbcc-b9b03101b3e9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") "
Sep 10 21:34:35.462204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462084    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-tmp-dir\") pod \"38b31228-5382-414c-bbcc-b9b03101b3e9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") "
Sep 10 21:34:35.462204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462131    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-home\") pod \"38b31228-5382-414c-bbcc-b9b03101b3e9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") "
Sep 10 21:34:35.462204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38b31228-5382-414c-bbcc-b9b03101b3e9-tls-certs\") pod \"38b31228-5382-414c-bbcc-b9b03101b3e9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") "
Sep 10 21:34:35.462370 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk476\" (UniqueName: \"kubernetes.io/projected/38b31228-5382-414c-bbcc-b9b03101b3e9-kube-api-access-dk476\") pod \"38b31228-5382-414c-bbcc-b9b03101b3e9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") "
Sep 10 21:34:35.462370 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462244    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-model-cache\") pod \"38b31228-5382-414c-bbcc-b9b03101b3e9\" (UID: \"38b31228-5382-414c-bbcc-b9b03101b3e9\") "
Sep 10 21:34:35.462467 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462398    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38b31228-5382-414c-bbcc-b9b03101b3e9" (UID: "38b31228-5382-414c-bbcc-b9b03101b3e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:35.462503 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462456    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-home" (OuterVolumeSpecName: "home") pod "38b31228-5382-414c-bbcc-b9b03101b3e9" (UID: "38b31228-5382-414c-bbcc-b9b03101b3e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:35.462601 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462582    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "38b31228-5382-414c-bbcc-b9b03101b3e9" (UID: "38b31228-5382-414c-bbcc-b9b03101b3e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:35.462680 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462666    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:35.462740 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462704    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:35.462740 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.462714    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:35.464383 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.464363    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-dshm" (OuterVolumeSpecName: "dshm") pod "38b31228-5382-414c-bbcc-b9b03101b3e9" (UID: "38b31228-5382-414c-bbcc-b9b03101b3e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:35.464549 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.464531    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38b31228-5382-414c-bbcc-b9b03101b3e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38b31228-5382-414c-bbcc-b9b03101b3e9" (UID: "38b31228-5382-414c-bbcc-b9b03101b3e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:34:35.464611 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.464584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b31228-5382-414c-bbcc-b9b03101b3e9-kube-api-access-dk476" (OuterVolumeSpecName: "kube-api-access-dk476") pod "38b31228-5382-414c-bbcc-b9b03101b3e9" (UID: "38b31228-5382-414c-bbcc-b9b03101b3e9"). InnerVolumeSpecName "kube-api-access-dk476". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:34:35.563787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.563750    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38b31228-5382-414c-bbcc-b9b03101b3e9-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:35.563787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.563779    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38b31228-5382-414c-bbcc-b9b03101b3e9-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:35.563787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.563792    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk476\" (UniqueName: \"kubernetes.io/projected/38b31228-5382-414c-bbcc-b9b03101b3e9-kube-api-access-dk476\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:35.862997 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.862953    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq" podUID="19e74f05-efff-470b-9ab9-91495b7e6251" containerName="main" containerID="cri-o://2c464f06700d55e1154c70f3cc518c5ff340f54849632af443e0ac0de5444b41" gracePeriod=45
Sep 10 21:34:35.990919 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.990888    2580 generic.go:356] "Generic (PLEG): container finished" podID="38b31228-5382-414c-bbcc-b9b03101b3e9" containerID="970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b" exitCode=0
Sep 10 21:34:35.991108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.990962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9" event={"ID":"38b31228-5382-414c-bbcc-b9b03101b3e9","Type":"ContainerDied","Data":"970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b"}
Sep 10 21:34:35.991108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.990993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9" event={"ID":"38b31228-5382-414c-bbcc-b9b03101b3e9","Type":"ContainerDied","Data":"b940a0e15478881544642f0e37200231da414cfb40260727801706cdecd6f97e"}
Sep 10 21:34:35.991108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.990962    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"
Sep 10 21:34:35.991108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.991063    2580 scope.go:117] "RemoveContainer" containerID="970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b"
Sep 10 21:34:35.992774 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.992749    2580 generic.go:356] "Generic (PLEG): container finished" podID="19e74f05-efff-470b-9ab9-91495b7e6251" containerID="2c464f06700d55e1154c70f3cc518c5ff340f54849632af443e0ac0de5444b41" exitCode=0
Sep 10 21:34:35.992995 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:35.992788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq" event={"ID":"19e74f05-efff-470b-9ab9-91495b7e6251","Type":"ContainerDied","Data":"2c464f06700d55e1154c70f3cc518c5ff340f54849632af443e0ac0de5444b41"}
Sep 10 21:34:36.000872 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.000854    2580 scope.go:117] "RemoveContainer" containerID="970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b"
Sep 10 21:34:36.001170 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:34:36.001153    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b\": container with ID starting with 970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b not found: ID does not exist" containerID="970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b"
Sep 10 21:34:36.001255 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.001176    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b"} err="failed to get container status \"970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b\": rpc error: code = NotFound desc = could not find container \"970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b\": container with ID starting with 970da02adbcdd7dc80773f129d51912833dc36c918c3baac199a8546f0eb1d0b not found: ID does not exist"
Sep 10 21:34:36.020742 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.020709    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"]
Sep 10 21:34:36.028075 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.028042    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-15ae234f-kserve-66c69cf8f6-v5vh9"]
Sep 10 21:34:36.112649 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.112622    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:34:36.170339 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-home\") pod \"19e74f05-efff-470b-9ab9-91495b7e6251\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") "
Sep 10 21:34:36.170793 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170354    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19e74f05-efff-470b-9ab9-91495b7e6251-tls-certs\") pod \"19e74f05-efff-470b-9ab9-91495b7e6251\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") "
Sep 10 21:34:36.170793 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-dshm\") pod \"19e74f05-efff-470b-9ab9-91495b7e6251\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") "
Sep 10 21:34:36.170793 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170452    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-tmp-dir\") pod \"19e74f05-efff-470b-9ab9-91495b7e6251\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") "
Sep 10 21:34:36.170793 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-model-cache\") pod \"19e74f05-efff-470b-9ab9-91495b7e6251\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") "
Sep 10 21:34:36.170793 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170544    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2q4h\" (UniqueName: \"kubernetes.io/projected/19e74f05-efff-470b-9ab9-91495b7e6251-kube-api-access-w2q4h\") pod \"19e74f05-efff-470b-9ab9-91495b7e6251\" (UID: \"19e74f05-efff-470b-9ab9-91495b7e6251\") "
Sep 10 21:34:36.170793 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-home" (OuterVolumeSpecName: "home") pod "19e74f05-efff-470b-9ab9-91495b7e6251" (UID: "19e74f05-efff-470b-9ab9-91495b7e6251"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:36.170793 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19e74f05-efff-470b-9ab9-91495b7e6251" (UID: "19e74f05-efff-470b-9ab9-91495b7e6251"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:36.170793 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170762    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-model-cache" (OuterVolumeSpecName: "model-cache") pod "19e74f05-efff-470b-9ab9-91495b7e6251" (UID: "19e74f05-efff-470b-9ab9-91495b7e6251"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:36.171316 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170897    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:36.171316 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170919    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:36.171316 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.170935    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:36.173523 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.173491    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-dshm" (OuterVolumeSpecName: "dshm") pod "19e74f05-efff-470b-9ab9-91495b7e6251" (UID: "19e74f05-efff-470b-9ab9-91495b7e6251"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:36.173660 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.173521    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e74f05-efff-470b-9ab9-91495b7e6251-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19e74f05-efff-470b-9ab9-91495b7e6251" (UID: "19e74f05-efff-470b-9ab9-91495b7e6251"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:34:36.173660 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.173576    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e74f05-efff-470b-9ab9-91495b7e6251-kube-api-access-w2q4h" (OuterVolumeSpecName: "kube-api-access-w2q4h") pod "19e74f05-efff-470b-9ab9-91495b7e6251" (UID: "19e74f05-efff-470b-9ab9-91495b7e6251"). InnerVolumeSpecName "kube-api-access-w2q4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:34:36.271673 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.271640    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19e74f05-efff-470b-9ab9-91495b7e6251-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:36.271673 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.271670    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19e74f05-efff-470b-9ab9-91495b7e6251-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:36.271673 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:36.271680    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2q4h\" (UniqueName: \"kubernetes.io/projected/19e74f05-efff-470b-9ab9-91495b7e6251-kube-api-access-w2q4h\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:37.000890 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:37.000860    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"
Sep 10 21:34:37.001070 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:37.000897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq" event={"ID":"19e74f05-efff-470b-9ab9-91495b7e6251","Type":"ContainerDied","Data":"c2bbf9ce97db978a60f1749620ad6c8e2986bf3ba1a2aea5e5ee597093b02234"}
Sep 10 21:34:37.001070 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:37.000940    2580 scope.go:117] "RemoveContainer" containerID="2c464f06700d55e1154c70f3cc518c5ff340f54849632af443e0ac0de5444b41"
Sep 10 21:34:37.022636 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:37.022604    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"]
Sep 10 21:34:37.027434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:37.027386    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-4b08e7/autores-model-4b08e7-kserve-54b7c576cf-b5pjq"]
Sep 10 21:34:37.821132 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:37.821087    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19e74f05-efff-470b-9ab9-91495b7e6251" path="/var/lib/kubelet/pods/19e74f05-efff-470b-9ab9-91495b7e6251/volumes"
Sep 10 21:34:37.821714 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:37.821693    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38b31228-5382-414c-bbcc-b9b03101b3e9" path="/var/lib/kubelet/pods/38b31228-5382-414c-bbcc-b9b03101b3e9/volumes"
Sep 10 21:34:38.736298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:38.736260    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:34:39.611386 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:39.611356    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"]
Sep 10 21:34:39.611848 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:39.611577    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy" containerID="cri-o://d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133" gracePeriod=30
Sep 10 21:34:40.208656 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.208625    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"]
Sep 10 21:34:40.209024 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209012    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38b31228-5382-414c-bbcc-b9b03101b3e9" containerName="main"
Sep 10 21:34:40.209074 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209026    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38b31228-5382-414c-bbcc-b9b03101b3e9" containerName="main"
Sep 10 21:34:40.209074 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209037    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e74f05-efff-470b-9ab9-91495b7e6251" containerName="main"
Sep 10 21:34:40.209074 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209042    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e74f05-efff-470b-9ab9-91495b7e6251" containerName="main"
Sep 10 21:34:40.209074 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209053    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="204e9152-d0bf-4a1d-988d-ad86c0f303f2" containerName="maas-api"
Sep 10 21:34:40.209074 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209059    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="204e9152-d0bf-4a1d-988d-ad86c0f303f2" containerName="maas-api"
Sep 10 21:34:40.209074 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209068    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4bf264-50b2-4161-b75f-1a577782dc30" containerName="payload-pre-processing"
Sep 10 21:34:40.209074 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209073    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4bf264-50b2-4161-b75f-1a577782dc30" containerName="payload-pre-processing"
Sep 10 21:34:40.209305 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209127    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="204e9152-d0bf-4a1d-988d-ad86c0f303f2" containerName="maas-api"
Sep 10 21:34:40.209305 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209135    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e74f05-efff-470b-9ab9-91495b7e6251" containerName="main"
Sep 10 21:34:40.209305 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209143    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="38b31228-5382-414c-bbcc-b9b03101b3e9" containerName="main"
Sep 10 21:34:40.209305 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.209149    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4bf264-50b2-4161-b75f-1a577782dc30" containerName="payload-pre-processing"
Sep 10 21:34:40.218171 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.218143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.221535 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.221512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5bba43\"/\"openshift-service-ca.crt\""
Sep 10 21:34:40.221667 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.221536    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5bba43\"/\"kube-root-ca.crt\""
Sep 10 21:34:40.222171 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.222131    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5bba43\"/\"default-dockercfg-dgjrv\""
Sep 10 21:34:40.222586 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.222565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5bba43\"/\"test-model-5bba43-kserve-self-signed-certs\""
Sep 10 21:34:40.225532 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.225301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"]
Sep 10 21:34:40.313592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.313553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-dshm\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.313787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.313625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbngc\" (UniqueName: \"kubernetes.io/projected/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-kube-api-access-hbngc\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.313787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.313664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-model-cache\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.313787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.313698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tls-certs\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.313787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.313720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tmp-dir\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.313931 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.313810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-home\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.414498 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.414462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-home\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.414699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.414513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-dshm\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.414699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.414561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbngc\" (UniqueName: \"kubernetes.io/projected/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-kube-api-access-hbngc\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.414699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.414583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-model-cache\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.414699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.414625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tls-certs\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.414699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.414655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tmp-dir\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.414973 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.414924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-home\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.415016 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.414986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-model-cache\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.415107 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.415085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tmp-dir\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.417043 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.417021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-dshm\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.417266 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.417250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tls-certs\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.422578 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.422555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbngc\" (UniqueName: \"kubernetes.io/projected/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-kube-api-access-hbngc\") pod \"test-model-5bba43-kserve-76cc567b7f-q48vj\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") " pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.528628 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.528519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:40.804067 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:40.804039    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"]
Sep 10 21:34:40.805495 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:34:40.805453    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c985a09_4f60_4d6a_a4ac_3a25f4432efc.slice/crio-05b294174a728f91562b5f2ad58512eda9addc6b16ac499c85b2d9e36ddfa86e WatchSource:0}: Error finding container 05b294174a728f91562b5f2ad58512eda9addc6b16ac499c85b2d9e36ddfa86e: Status 404 returned error can't find the container with id 05b294174a728f91562b5f2ad58512eda9addc6b16ac499c85b2d9e36ddfa86e
Sep 10 21:34:41.021674 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:41.021637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj" event={"ID":"5c985a09-4f60-4d6a-a4ac-3a25f4432efc","Type":"ContainerStarted","Data":"cd0facaf77ec9edbf2db62a50aa30aaa14967a3804fd768f3ca6dfeddb7d185c"}
Sep 10 21:34:41.021835 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:41.021690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj" event={"ID":"5c985a09-4f60-4d6a-a4ac-3a25f4432efc","Type":"ContainerStarted","Data":"05b294174a728f91562b5f2ad58512eda9addc6b16ac499c85b2d9e36ddfa86e"}
Sep 10 21:34:41.042612 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:41.042563    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj" podStartSLOduration=1.042547215 podStartE2EDuration="1.042547215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:34:41.041495162 +0000 UTC m=+919.689409513" watchObservedRunningTime="2026-09-10 21:34:41.042547215 +0000 UTC m=+919.690461382"
Sep 10 21:34:42.333780 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.333740    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz" podUID="1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" containerName="main" containerID="cri-o://1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d" gracePeriod=45
Sep 10 21:34:42.598879 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.598817    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:34:42.738824 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.738784    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-home\") pod \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") "
Sep 10 21:34:42.738994 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.738833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tls-certs\") pod \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") "
Sep 10 21:34:42.738994 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.738864    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tmp-dir\") pod \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") "
Sep 10 21:34:42.738994 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.738885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-model-cache\") pod \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") "
Sep 10 21:34:42.738994 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.738908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tftb6\" (UniqueName: \"kubernetes.io/projected/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-kube-api-access-tftb6\") pod \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") "
Sep 10 21:34:42.738994 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.738942    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-dshm\") pod \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\" (UID: \"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d\") "
Sep 10 21:34:42.739269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.739155    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-home" (OuterVolumeSpecName: "home") pod "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" (UID: "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:42.739269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.739162    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" (UID: "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:42.739269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.739222    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" (UID: "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:42.739446 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.739292    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:42.739446 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.739303    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:42.739446 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.739312    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:42.741255 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.741232    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-dshm" (OuterVolumeSpecName: "dshm") pod "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" (UID: "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:42.741429 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.741264    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" (UID: "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:34:42.741429 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.741336    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-kube-api-access-tftb6" (OuterVolumeSpecName: "kube-api-access-tftb6") pod "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" (UID: "1b35199a-3ac9-4efa-bf8e-50b735b6ce7d"). InnerVolumeSpecName "kube-api-access-tftb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:34:42.840233 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.840203    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:42.840233 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.840232    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tftb6\" (UniqueName: \"kubernetes.io/projected/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-kube-api-access-tftb6\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:42.840448 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:42.840244    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:43.032890 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:43.032857    2580 generic.go:356] "Generic (PLEG): container finished" podID="1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" containerID="1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d" exitCode=0
Sep 10 21:34:43.033088 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:43.032900    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"
Sep 10 21:34:43.033088 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:43.032928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz" event={"ID":"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d","Type":"ContainerDied","Data":"1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d"}
Sep 10 21:34:43.033088 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:43.032966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz" event={"ID":"1b35199a-3ac9-4efa-bf8e-50b735b6ce7d","Type":"ContainerDied","Data":"8a51114c48f9c1591e235669a0ab9b1617b808e05d8c803bd3bac149cb6828aa"}
Sep 10 21:34:43.033088 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:43.032989    2580 scope.go:117] "RemoveContainer" containerID="1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d"
Sep 10 21:34:43.041899 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:43.041882    2580 scope.go:117] "RemoveContainer" containerID="1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d"
Sep 10 21:34:43.042152 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:34:43.042133    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d\": container with ID starting with 1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d not found: ID does not exist" containerID="1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d"
Sep 10 21:34:43.042189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:43.042163    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d"} err="failed to get container status \"1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d\": rpc error: code = NotFound desc = could not find container \"1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d\": container with ID starting with 1ca1c72514078fbd18ee33f96545daaed804538a24fffca210f02f895e32a06d not found: ID does not exist"
Sep 10 21:34:43.055313 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:43.055281    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"]
Sep 10 21:34:43.061237 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:43.061206    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-15ae234f-kserve-7d48dd9846-4rszz"]
Sep 10 21:34:43.821041 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:43.821007    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" path="/var/lib/kubelet/pods/1b35199a-3ac9-4efa-bf8e-50b735b6ce7d/volumes"
Sep 10 21:34:44.864264 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.864239    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:34:44.959831 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.959801    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994645f5-5899-4c60-ab5e-1443c2262811-istio-podinfo\") pod \"994645f5-5899-4c60-ab5e-1443c2262811\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") "
Sep 10 21:34:44.960002 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.959856    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994645f5-5899-4c60-ab5e-1443c2262811-istiod-ca-cert\") pod \"994645f5-5899-4c60-ab5e-1443c2262811\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") "
Sep 10 21:34:44.960002 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.959966    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j8zb\" (UniqueName: \"kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-kube-api-access-8j8zb\") pod \"994645f5-5899-4c60-ab5e-1443c2262811\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") "
Sep 10 21:34:44.960092 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960006    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-credential-socket\") pod \"994645f5-5899-4c60-ab5e-1443c2262811\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") "
Sep 10 21:34:44.960092 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-certs\") pod \"994645f5-5899-4c60-ab5e-1443c2262811\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") "
Sep 10 21:34:44.960092 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960089    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-envoy\") pod \"994645f5-5899-4c60-ab5e-1443c2262811\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") "
Sep 10 21:34:44.960233 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960129    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-data\") pod \"994645f5-5899-4c60-ab5e-1443c2262811\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") "
Sep 10 21:34:44.960233 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960157    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-socket\") pod \"994645f5-5899-4c60-ab5e-1443c2262811\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") "
Sep 10 21:34:44.960233 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960197    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-istio-token\") pod \"994645f5-5899-4c60-ab5e-1443c2262811\" (UID: \"994645f5-5899-4c60-ab5e-1443c2262811\") "
Sep 10 21:34:44.960381 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960278    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "994645f5-5899-4c60-ab5e-1443c2262811" (UID: "994645f5-5899-4c60-ab5e-1443c2262811"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:44.960381 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960197    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994645f5-5899-4c60-ab5e-1443c2262811-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "994645f5-5899-4c60-ab5e-1443c2262811" (UID: "994645f5-5899-4c60-ab5e-1443c2262811"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:34:44.960529 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "994645f5-5899-4c60-ab5e-1443c2262811" (UID: "994645f5-5899-4c60-ab5e-1443c2262811"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:44.960529 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960520    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-data" (OuterVolumeSpecName: "istio-data") pod "994645f5-5899-4c60-ab5e-1443c2262811" (UID: "994645f5-5899-4c60-ab5e-1443c2262811"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:44.960635 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960613    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "994645f5-5899-4c60-ab5e-1443c2262811" (UID: "994645f5-5899-4c60-ab5e-1443c2262811"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:44.960756 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960697    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:44.960899 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960836    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:44.960899 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960900    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:44.961089 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.960913    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994645f5-5899-4c60-ab5e-1443c2262811-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:44.962334 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.962310    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-kube-api-access-8j8zb" (OuterVolumeSpecName: "kube-api-access-8j8zb") pod "994645f5-5899-4c60-ab5e-1443c2262811" (UID: "994645f5-5899-4c60-ab5e-1443c2262811"). InnerVolumeSpecName "kube-api-access-8j8zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:34:44.962334 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.962317    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/994645f5-5899-4c60-ab5e-1443c2262811-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "994645f5-5899-4c60-ab5e-1443c2262811" (UID: "994645f5-5899-4c60-ab5e-1443c2262811"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:34:44.962497 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.962361    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "994645f5-5899-4c60-ab5e-1443c2262811" (UID: "994645f5-5899-4c60-ab5e-1443c2262811"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:34:44.962497 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:44.962467    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-istio-token" (OuterVolumeSpecName: "istio-token") pod "994645f5-5899-4c60-ab5e-1443c2262811" (UID: "994645f5-5899-4c60-ab5e-1443c2262811"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:34:45.027915 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.027879    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"]
Sep 10 21:34:45.028518 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.028497    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" containerName="main"
Sep 10 21:34:45.028518 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.028517    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" containerName="main"
Sep 10 21:34:45.028699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.028546    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy"
Sep 10 21:34:45.028699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.028555    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy"
Sep 10 21:34:45.028699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.028571    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy"
Sep 10 21:34:45.028699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.028580    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy"
Sep 10 21:34:45.028699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.028665    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b35199a-3ac9-4efa-bf8e-50b735b6ce7d" containerName="main"
Sep 10 21:34:45.028699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.028675    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy"
Sep 10 21:34:45.028699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.028686    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="994645f5-5899-4c60-ab5e-1443c2262811" containerName="istio-proxy"
Sep 10 21:34:45.032312 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.032290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.036122 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.036094    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-610c0f-serving-cert\""
Sep 10 21:34:45.045872 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.045810    2580 generic.go:356] "Generic (PLEG): container finished" podID="994645f5-5899-4c60-ab5e-1443c2262811" containerID="d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133" exitCode=0
Sep 10 21:34:45.045872 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.045846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" event={"ID":"994645f5-5899-4c60-ab5e-1443c2262811","Type":"ContainerDied","Data":"d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133"}
Sep 10 21:34:45.046061 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.045877    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"
Sep 10 21:34:45.046061 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.045903    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf" event={"ID":"994645f5-5899-4c60-ab5e-1443c2262811","Type":"ContainerDied","Data":"b03b0dd5c0973de79dd24bbbcdf5bbc99605765ba91e56435828736cae770380"}
Sep 10 21:34:45.046061 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.045934    2580 scope.go:117] "RemoveContainer" containerID="d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133"
Sep 10 21:34:45.055024 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.055002    2580 scope.go:117] "RemoveContainer" containerID="cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7"
Sep 10 21:34:45.061431 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.061389    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8zb\" (UniqueName: \"kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-kube-api-access-8j8zb\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:45.061540 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.061434    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:45.061540 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.061447    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994645f5-5899-4c60-ab5e-1443c2262811-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:45.061540 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.061456    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994645f5-5899-4c60-ab5e-1443c2262811-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:45.061540 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.061464    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994645f5-5899-4c60-ab5e-1443c2262811-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:34:45.063261 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.063239    2580 scope.go:117] "RemoveContainer" containerID="d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133"
Sep 10 21:34:45.063545 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:34:45.063528    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133\": container with ID starting with d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133 not found: ID does not exist" containerID="d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133"
Sep 10 21:34:45.063604 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.063553    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133"} err="failed to get container status \"d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133\": rpc error: code = NotFound desc = could not find container \"d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133\": container with ID starting with d2b9f36e86f4700af60e59a2ead99a67f6f738b0062dd997680f51a0262f3133 not found: ID does not exist"
Sep 10 21:34:45.063604 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.063569    2580 scope.go:117] "RemoveContainer" containerID="cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7"
Sep 10 21:34:45.063845 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:34:45.063827    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7\": container with ID starting with cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7 not found: ID does not exist" containerID="cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7"
Sep 10 21:34:45.063883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.063851    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7"} err="failed to get container status \"cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7\": rpc error: code = NotFound desc = could not find container \"cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7\": container with ID starting with cc26faff90545bab2101970baaa706cb03937e67bb91e37992fd3bb789218bf7 not found: ID does not exist"
Sep 10 21:34:45.068668 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.068645    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"]
Sep 10 21:34:45.112926 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.112897    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"]
Sep 10 21:34:45.124801 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.124780    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4b08e7-openshift-default-79dd656bcd-lwzvf"]
Sep 10 21:34:45.162471 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.162445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vfb\" (UniqueName: \"kubernetes.io/projected/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-kube-api-access-j9vfb\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.162607 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.162479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-maas-api-tls\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.162607 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.162522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-openshift-service-ca\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.162607 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.162556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-metrics-certs\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.263795 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.263710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vfb\" (UniqueName: \"kubernetes.io/projected/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-kube-api-access-j9vfb\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.263795 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.263752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-maas-api-tls\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.263795 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.263793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-openshift-service-ca\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.264031 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.263839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-metrics-certs\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.264031 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:34:45.263933    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-serving-cert: secret "maas-api-e2e-explicit-610c0f-serving-cert" not found
Sep 10 21:34:45.264031 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:34:45.263999    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-maas-api-tls podName:8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742 nodeName:}" failed. No retries permitted until 2026-09-10 21:34:45.763984163 +0000 UTC m=+924.411898309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-maas-api-tls") pod "maas-api-e2e-explicit-610c0f-555596b794-4gwtj" (UID: "8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742") : secret "maas-api-e2e-explicit-610c0f-serving-cert" not found
Sep 10 21:34:45.264500 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.264482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-openshift-service-ca\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.266386 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.266368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-metrics-certs\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.276679 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.276656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vfb\" (UniqueName: \"kubernetes.io/projected/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-kube-api-access-j9vfb\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.769749 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.769709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-maas-api-tls\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.772315 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.772296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-maas-api-tls\") pod \"maas-api-e2e-explicit-610c0f-555596b794-4gwtj\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:45.821194 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.821164    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="994645f5-5899-4c60-ab5e-1443c2262811" path="/var/lib/kubelet/pods/994645f5-5899-4c60-ab5e-1443c2262811/volumes"
Sep 10 21:34:45.942724 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:45.942685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:46.155027 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.154930    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"]
Sep 10 21:34:46.159621 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.159600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:34:46.162338 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.162318    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-610c0f-dockercfg-4n2gr\""
Sep 10 21:34:46.162473 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.162323    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-610c0f\""
Sep 10 21:34:46.170368 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.170347    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"]
Sep 10 21:34:46.229666 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.229582    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"]
Sep 10 21:34:46.232104 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:34:46.232080    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3cf6e7_2d33_4b23_a31e_1ccf9e27a742.slice/crio-05d909717f2c21fe1e6ab4c5612c477ec20cd336f9d3c3c54fb6ade33a03d080 WatchSource:0}: Error finding container 05d909717f2c21fe1e6ab4c5612c477ec20cd336f9d3c3c54fb6ade33a03d080: Status 404 returned error can't find the container with id 05d909717f2c21fe1e6ab4c5612c477ec20cd336f9d3c3c54fb6ade33a03d080
Sep 10 21:34:46.275579 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.275556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f025322-e837-42c1-8254-12399f40fb82-plugins-config-volume\") pod \"payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64\" (UID: \"3f025322-e837-42c1-8254-12399f40fb82\") " pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:34:46.275687 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.275591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9s7w\" (UniqueName: \"kubernetes.io/projected/3f025322-e837-42c1-8254-12399f40fb82-kube-api-access-r9s7w\") pod \"payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64\" (UID: \"3f025322-e837-42c1-8254-12399f40fb82\") " pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:34:46.376308 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.376275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f025322-e837-42c1-8254-12399f40fb82-plugins-config-volume\") pod \"payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64\" (UID: \"3f025322-e837-42c1-8254-12399f40fb82\") " pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:34:46.376433 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.376314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9s7w\" (UniqueName: \"kubernetes.io/projected/3f025322-e837-42c1-8254-12399f40fb82-kube-api-access-r9s7w\") pod \"payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64\" (UID: \"3f025322-e837-42c1-8254-12399f40fb82\") " pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:34:46.377040 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.377023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f025322-e837-42c1-8254-12399f40fb82-plugins-config-volume\") pod \"payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64\" (UID: \"3f025322-e837-42c1-8254-12399f40fb82\") " pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:34:46.386626 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.386596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9s7w\" (UniqueName: \"kubernetes.io/projected/3f025322-e837-42c1-8254-12399f40fb82-kube-api-access-r9s7w\") pod \"payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64\" (UID: \"3f025322-e837-42c1-8254-12399f40fb82\") " pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:34:46.471214 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.471183    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:34:46.757627 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:46.757479    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"]
Sep 10 21:34:46.760395 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:34:46.760364    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f025322_e837_42c1_8254_12399f40fb82.slice/crio-07afba34785415d36328a3b3844edb4c5b12c5bab00aa864b1843c7ce56142fd WatchSource:0}: Error finding container 07afba34785415d36328a3b3844edb4c5b12c5bab00aa864b1843c7ce56142fd: Status 404 returned error can't find the container with id 07afba34785415d36328a3b3844edb4c5b12c5bab00aa864b1843c7ce56142fd
Sep 10 21:34:47.058406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:47.058316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj" event={"ID":"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742","Type":"ContainerStarted","Data":"aa6f1c226fa8c63947c8ee0a4ba3ab4f868a9fccf5cc0d584f496ec846b9431f"}
Sep 10 21:34:47.058406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:47.058368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj" event={"ID":"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742","Type":"ContainerStarted","Data":"05d909717f2c21fe1e6ab4c5612c477ec20cd336f9d3c3c54fb6ade33a03d080"}
Sep 10 21:34:47.058938 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:47.058454    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:47.060265 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:47.060244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64" event={"ID":"3f025322-e837-42c1-8254-12399f40fb82","Type":"ContainerStarted","Data":"aa773296d858bd32964c689576c9c2bbb768b756dcd1e8223d8c6e0799b15f96"}
Sep 10 21:34:47.060394 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:47.060282    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:34:47.060394 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:47.060299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64" event={"ID":"3f025322-e837-42c1-8254-12399f40fb82","Type":"ContainerStarted","Data":"07afba34785415d36328a3b3844edb4c5b12c5bab00aa864b1843c7ce56142fd"}
Sep 10 21:34:47.084012 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:47.083794    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj" podStartSLOduration=2.6867399929999998 podStartE2EDuration="3.083778278s" totalImagesPullingTime="397.038285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:34:47.082324433 +0000 UTC m=+925.730238602" watchObservedRunningTime="2026-09-10 21:34:47.083778278 +0000 UTC m=+925.731692446"
Sep 10 21:34:47.113809 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:47.113768    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64" podStartSLOduration=1.11375481 podStartE2EDuration="1.11375481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:34:47.112906634 +0000 UTC m=+925.760820801" watchObservedRunningTime="2026-09-10 21:34:47.11375481 +0000 UTC m=+925.761668979"
Sep 10 21:34:50.529072 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:50.529030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:50.529072 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:50.529081    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:50.541533 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:50.541503    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:51.089407 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:51.089379    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:34:53.070543 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:53.070512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:34:56.732683 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:56.732642    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:34:58.067788 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:34:58.067756    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:35:13.914142 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:13.914108    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:35:17.307709 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.307680    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"]
Sep 10 21:35:17.310993 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.310970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.313531 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.313502    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-610c0f\"/\"kube-root-ca.crt\""
Sep 10 21:35:17.314110 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.314089    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-610c0f\"/\"openshift-service-ca.crt\""
Sep 10 21:35:17.314207 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.314104    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-610c0f\"/\"default-dockercfg-qth7p\""
Sep 10 21:35:17.315081 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.315057    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-610c0f\"/\"explicit-model-610c0f-kserve-self-signed-certs\""
Sep 10 21:35:17.323374 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.323350    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"]
Sep 10 21:35:17.464908 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.464865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436535af-796a-40db-a423-2aba0a88c3f3-tls-certs\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.465088 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.464965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-dshm\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.465088 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.464995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-tmp-dir\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.465088 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.465066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-model-cache\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.465218 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.465125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5m26\" (UniqueName: \"kubernetes.io/projected/436535af-796a-40db-a423-2aba0a88c3f3-kube-api-access-t5m26\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.465218 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.465152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-home\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.566407 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.566309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-dshm\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.566407 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.566355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-tmp-dir\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.566407 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.566402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-model-cache\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.566688 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.566467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5m26\" (UniqueName: \"kubernetes.io/projected/436535af-796a-40db-a423-2aba0a88c3f3-kube-api-access-t5m26\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.566688 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.566492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-home\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.566688 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.566555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436535af-796a-40db-a423-2aba0a88c3f3-tls-certs\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.566853 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.566749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-tmp-dir\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.566853 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.566807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-model-cache\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.566934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.566878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-home\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.568966 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.568938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-dshm\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.569154 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.569135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436535af-796a-40db-a423-2aba0a88c3f3-tls-certs\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.574214 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.574185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5m26\" (UniqueName: \"kubernetes.io/projected/436535af-796a-40db-a423-2aba0a88c3f3-kube-api-access-t5m26\") pod \"explicit-model-610c0f-kserve-6fd5846d5c-9sg62\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") " pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.621328 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.621286    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:17.898679 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:17.898655    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"]
Sep 10 21:35:17.901346 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:35:17.901320    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod436535af_796a_40db_a423_2aba0a88c3f3.slice/crio-196f2181b76a4645647e39e3758f2fe9ab4bd58c2445be944b71c2629586bc32 WatchSource:0}: Error finding container 196f2181b76a4645647e39e3758f2fe9ab4bd58c2445be944b71c2629586bc32: Status 404 returned error can't find the container with id 196f2181b76a4645647e39e3758f2fe9ab4bd58c2445be944b71c2629586bc32
Sep 10 21:35:18.213972 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:18.213936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62" event={"ID":"436535af-796a-40db-a423-2aba0a88c3f3","Type":"ContainerStarted","Data":"faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34"}
Sep 10 21:35:18.214160 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:18.213978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62" event={"ID":"436535af-796a-40db-a423-2aba0a88c3f3","Type":"ContainerStarted","Data":"196f2181b76a4645647e39e3758f2fe9ab4bd58c2445be944b71c2629586bc32"}
Sep 10 21:35:18.230605 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:18.230468    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62" podStartSLOduration=1.230448124 podStartE2EDuration="1.230448124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:35:18.230063899 +0000 UTC m=+956.877978179" watchObservedRunningTime="2026-09-10 21:35:18.230448124 +0000 UTC m=+956.878362294"
Sep 10 21:35:23.454344 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:23.454313    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"]
Sep 10 21:35:27.622234 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:27.622188    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:29.806600 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:29.806560    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:35:34.001770 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.001734    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"]
Sep 10 21:35:34.002195 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.001975    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj" podUID="8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742" containerName="maas-api" containerID="cri-o://aa6f1c226fa8c63947c8ee0a4ba3ab4f868a9fccf5cc0d584f496ec846b9431f" gracePeriod=30
Sep 10 21:35:34.063700 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.063667    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"]
Sep 10 21:35:34.063984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.063925    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64" podUID="3f025322-e837-42c1-8254-12399f40fb82" containerName="payload-processing" containerID="cri-o://aa773296d858bd32964c689576c9c2bbb768b756dcd1e8223d8c6e0799b15f96" gracePeriod=30
Sep 10 21:35:34.294967 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.294932    2580 generic.go:356] "Generic (PLEG): container finished" podID="3f025322-e837-42c1-8254-12399f40fb82" containerID="aa773296d858bd32964c689576c9c2bbb768b756dcd1e8223d8c6e0799b15f96" exitCode=0
Sep 10 21:35:34.295102 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.294997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64" event={"ID":"3f025322-e837-42c1-8254-12399f40fb82","Type":"ContainerDied","Data":"aa773296d858bd32964c689576c9c2bbb768b756dcd1e8223d8c6e0799b15f96"}
Sep 10 21:35:34.296624 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.296601    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742" containerID="aa6f1c226fa8c63947c8ee0a4ba3ab4f868a9fccf5cc0d584f496ec846b9431f" exitCode=0
Sep 10 21:35:34.296749 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.296641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj" event={"ID":"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742","Type":"ContainerDied","Data":"aa6f1c226fa8c63947c8ee0a4ba3ab4f868a9fccf5cc0d584f496ec846b9431f"}
Sep 10 21:35:34.313675 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.313655    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:35:34.342434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.341263    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-metrics-certs\") pod \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") "
Sep 10 21:35:34.342434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.341320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-openshift-service-ca\") pod \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") "
Sep 10 21:35:34.342434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.341407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-maas-api-tls\") pod \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") "
Sep 10 21:35:34.342434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.341480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9vfb\" (UniqueName: \"kubernetes.io/projected/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-kube-api-access-j9vfb\") pod \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\" (UID: \"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742\") "
Sep 10 21:35:34.345438 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.344993    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742" (UID: "8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:35:34.349327 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.349264    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742" (UID: "8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:35:34.349751 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.349722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-kube-api-access-j9vfb" (OuterVolumeSpecName: "kube-api-access-j9vfb") pod "8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742" (UID: "8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742"). InnerVolumeSpecName "kube-api-access-j9vfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:35:34.349871 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.349813    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742" (UID: "8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:35:34.364909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.364881    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:35:34.442993 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.442958    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f025322-e837-42c1-8254-12399f40fb82-plugins-config-volume\") pod \"3f025322-e837-42c1-8254-12399f40fb82\" (UID: \"3f025322-e837-42c1-8254-12399f40fb82\") "
Sep 10 21:35:34.443173 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.443067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9s7w\" (UniqueName: \"kubernetes.io/projected/3f025322-e837-42c1-8254-12399f40fb82-kube-api-access-r9s7w\") pod \"3f025322-e837-42c1-8254-12399f40fb82\" (UID: \"3f025322-e837-42c1-8254-12399f40fb82\") "
Sep 10 21:35:34.443306 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.443281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f025322-e837-42c1-8254-12399f40fb82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f025322-e837-42c1-8254-12399f40fb82" (UID: "3f025322-e837-42c1-8254-12399f40fb82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:35:34.443373 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.443354    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-openshift-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:34.443476 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.443379    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-maas-api-tls\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:34.443476 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.443393    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9vfb\" (UniqueName: \"kubernetes.io/projected/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-kube-api-access-j9vfb\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:34.443476 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.443444    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742-metrics-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:34.445339 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.445320    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f025322-e837-42c1-8254-12399f40fb82-kube-api-access-r9s7w" (OuterVolumeSpecName: "kube-api-access-r9s7w") pod "3f025322-e837-42c1-8254-12399f40fb82" (UID: "3f025322-e837-42c1-8254-12399f40fb82"). InnerVolumeSpecName "kube-api-access-r9s7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:35:34.544647 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.544567    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f025322-e837-42c1-8254-12399f40fb82-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:34.544647 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:34.544595    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9s7w\" (UniqueName: \"kubernetes.io/projected/3f025322-e837-42c1-8254-12399f40fb82-kube-api-access-r9s7w\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:35.302736 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.302646    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"
Sep 10 21:35:35.302736 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.302686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64" event={"ID":"3f025322-e837-42c1-8254-12399f40fb82","Type":"ContainerDied","Data":"07afba34785415d36328a3b3844edb4c5b12c5bab00aa864b1843c7ce56142fd"}
Sep 10 21:35:35.303259 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.302742    2580 scope.go:117] "RemoveContainer" containerID="aa773296d858bd32964c689576c9c2bbb768b756dcd1e8223d8c6e0799b15f96"
Sep 10 21:35:35.304148 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.304120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj" event={"ID":"8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742","Type":"ContainerDied","Data":"05d909717f2c21fe1e6ab4c5612c477ec20cd336f9d3c3c54fb6ade33a03d080"}
Sep 10 21:35:35.304221 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.304146    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"
Sep 10 21:35:35.311645 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.311503    2580 scope.go:117] "RemoveContainer" containerID="aa6f1c226fa8c63947c8ee0a4ba3ab4f868a9fccf5cc0d584f496ec846b9431f"
Sep 10 21:35:35.326530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.326503    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"]
Sep 10 21:35:35.330711 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.330686    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-610c0f-555596b794-4gwtj"]
Sep 10 21:35:35.341945 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.341910    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"]
Sep 10 21:35:35.346060 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.346034    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-610c0f-6f9bf9c7c-5rf64"]
Sep 10 21:35:35.821026 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.820992    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f025322-e837-42c1-8254-12399f40fb82" path="/var/lib/kubelet/pods/3f025322-e837-42c1-8254-12399f40fb82/volumes"
Sep 10 21:35:35.821402 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:35.821389    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742" path="/var/lib/kubelet/pods/8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742/volumes"
Sep 10 21:35:38.474277 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.474233    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62" podUID="436535af-796a-40db-a423-2aba0a88c3f3" containerName="main" containerID="cri-o://faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34" gracePeriod=45
Sep 10 21:35:38.722954 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.722932    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:38.784844 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.784766    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5m26\" (UniqueName: \"kubernetes.io/projected/436535af-796a-40db-a423-2aba0a88c3f3-kube-api-access-t5m26\") pod \"436535af-796a-40db-a423-2aba0a88c3f3\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") "
Sep 10 21:35:38.784844 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.784804    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436535af-796a-40db-a423-2aba0a88c3f3-tls-certs\") pod \"436535af-796a-40db-a423-2aba0a88c3f3\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") "
Sep 10 21:35:38.784844 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.784833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-model-cache\") pod \"436535af-796a-40db-a423-2aba0a88c3f3\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") "
Sep 10 21:35:38.785111 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.784854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-dshm\") pod \"436535af-796a-40db-a423-2aba0a88c3f3\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") "
Sep 10 21:35:38.785111 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.784882    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-tmp-dir\") pod \"436535af-796a-40db-a423-2aba0a88c3f3\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") "
Sep 10 21:35:38.785111 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.784962    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-home\") pod \"436535af-796a-40db-a423-2aba0a88c3f3\" (UID: \"436535af-796a-40db-a423-2aba0a88c3f3\") "
Sep 10 21:35:38.785279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.785126    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "436535af-796a-40db-a423-2aba0a88c3f3" (UID: "436535af-796a-40db-a423-2aba0a88c3f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:35:38.785279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.785236    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:38.785279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.785265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-home" (OuterVolumeSpecName: "home") pod "436535af-796a-40db-a423-2aba0a88c3f3" (UID: "436535af-796a-40db-a423-2aba0a88c3f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:35:38.785389 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.785273    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "436535af-796a-40db-a423-2aba0a88c3f3" (UID: "436535af-796a-40db-a423-2aba0a88c3f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:35:38.787212 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.787192    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-dshm" (OuterVolumeSpecName: "dshm") pod "436535af-796a-40db-a423-2aba0a88c3f3" (UID: "436535af-796a-40db-a423-2aba0a88c3f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:35:38.787287 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.787250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436535af-796a-40db-a423-2aba0a88c3f3-kube-api-access-t5m26" (OuterVolumeSpecName: "kube-api-access-t5m26") pod "436535af-796a-40db-a423-2aba0a88c3f3" (UID: "436535af-796a-40db-a423-2aba0a88c3f3"). InnerVolumeSpecName "kube-api-access-t5m26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:35:38.787287 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.787268    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/436535af-796a-40db-a423-2aba0a88c3f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "436535af-796a-40db-a423-2aba0a88c3f3" (UID: "436535af-796a-40db-a423-2aba0a88c3f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:35:38.886034 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.886014    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:38.886034 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.886035    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5m26\" (UniqueName: \"kubernetes.io/projected/436535af-796a-40db-a423-2aba0a88c3f3-kube-api-access-t5m26\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:38.886180 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.886046    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436535af-796a-40db-a423-2aba0a88c3f3-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:38.886180 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.886055    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:38.886180 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:38.886081    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436535af-796a-40db-a423-2aba0a88c3f3-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:35:39.328871 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:39.328831    2580 generic.go:356] "Generic (PLEG): container finished" podID="436535af-796a-40db-a423-2aba0a88c3f3" containerID="faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34" exitCode=0
Sep 10 21:35:39.328871 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:39.328871    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"
Sep 10 21:35:39.329121 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:39.328916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62" event={"ID":"436535af-796a-40db-a423-2aba0a88c3f3","Type":"ContainerDied","Data":"faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34"}
Sep 10 21:35:39.329121 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:39.328961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62" event={"ID":"436535af-796a-40db-a423-2aba0a88c3f3","Type":"ContainerDied","Data":"196f2181b76a4645647e39e3758f2fe9ab4bd58c2445be944b71c2629586bc32"}
Sep 10 21:35:39.329121 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:39.328981    2580 scope.go:117] "RemoveContainer" containerID="faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34"
Sep 10 21:35:39.338583 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:39.338559    2580 scope.go:117] "RemoveContainer" containerID="faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34"
Sep 10 21:35:39.338834 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:35:39.338816    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34\": container with ID starting with faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34 not found: ID does not exist" containerID="faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34"
Sep 10 21:35:39.338886 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:39.338843    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34"} err="failed to get container status \"faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34\": rpc error: code = NotFound desc = could not find container \"faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34\": container with ID starting with faa86c818df7beee6c91e7c7e51e001d6d27d5922142206a70530b492b5ffd34 not found: ID does not exist"
Sep 10 21:35:39.350527 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:39.350503    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"]
Sep 10 21:35:39.354448 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:39.354404    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-610c0f/explicit-model-610c0f-kserve-6fd5846d5c-9sg62"]
Sep 10 21:35:39.820317 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:39.820281    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="436535af-796a-40db-a423-2aba0a88c3f3" path="/var/lib/kubelet/pods/436535af-796a-40db-a423-2aba0a88c3f3/volumes"
Sep 10 21:35:45.616923 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:45.616886    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:35:57.608576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:35:57.608539    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:36:07.821745 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.821705    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"]
Sep 10 21:36:07.822281 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.822219    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742" containerName="maas-api"
Sep 10 21:36:07.822281 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.822238    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742" containerName="maas-api"
Sep 10 21:36:07.822281 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.822269    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f025322-e837-42c1-8254-12399f40fb82" containerName="payload-processing"
Sep 10 21:36:07.822281 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.822277    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f025322-e837-42c1-8254-12399f40fb82" containerName="payload-processing"
Sep 10 21:36:07.822537 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.822296    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="436535af-796a-40db-a423-2aba0a88c3f3" containerName="main"
Sep 10 21:36:07.822537 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.822305    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="436535af-796a-40db-a423-2aba0a88c3f3" containerName="main"
Sep 10 21:36:07.822537 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.822406    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="436535af-796a-40db-a423-2aba0a88c3f3" containerName="main"
Sep 10 21:36:07.822537 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.822440    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c3cf6e7-2d33-4b23-a31e-1ccf9e27a742" containerName="maas-api"
Sep 10 21:36:07.822537 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.822453    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f025322-e837-42c1-8254-12399f40fb82" containerName="payload-processing"
Sep 10 21:36:07.825914 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.825889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:07.826969 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.826945    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"]
Sep 10 21:36:07.828232 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.828215    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-295803\"/\"openshift-service-ca.crt\""
Sep 10 21:36:07.828923 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.828906    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-295803\"/\"default-dockercfg-xbw6w\""
Sep 10 21:36:07.829126 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.828951    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-295803\"/\"noten-model-295803-kserve-self-signed-certs\""
Sep 10 21:36:07.829240 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.828954    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-295803\"/\"kube-root-ca.crt\""
Sep 10 21:36:07.957826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.957788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-home\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:07.958006 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.957840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-dshm\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:07.958006 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.957860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-tmp-dir\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:07.958006 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.957891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4srtz\" (UniqueName: \"kubernetes.io/projected/cf53bd83-4d28-438f-b690-a2a136147405-kube-api-access-4srtz\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:07.958006 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.957913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-model-cache\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:07.958006 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:07.957935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf53bd83-4d28-438f-b690-a2a136147405-tls-certs\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.058758 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.058723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf53bd83-4d28-438f-b690-a2a136147405-tls-certs\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.058934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.058809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-home\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.058934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.058848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-dshm\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.058934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.058866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-tmp-dir\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.058934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.058888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srtz\" (UniqueName: \"kubernetes.io/projected/cf53bd83-4d28-438f-b690-a2a136147405-kube-api-access-4srtz\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.058934 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.058904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-model-cache\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.059317 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.059284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-tmp-dir\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.059317 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.059315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-model-cache\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.059509 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.059376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-home\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.061269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.061245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-dshm\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.061507 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.061490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf53bd83-4d28-438f-b690-a2a136147405-tls-certs\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.067150 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.067124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srtz\" (UniqueName: \"kubernetes.io/projected/cf53bd83-4d28-438f-b690-a2a136147405-kube-api-access-4srtz\") pod \"noten-model-295803-kserve-7c47b99475-58f8q\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") " pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.137535 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.137440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:08.409746 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.409722    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"]
Sep 10 21:36:08.411855 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:36:08.411827    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf53bd83_4d28_438f_b690_a2a136147405.slice/crio-703babb3dc0a96fd4517ff32b65a00bf9c95fd96205fa24f40f196149fbcb8e7 WatchSource:0}: Error finding container 703babb3dc0a96fd4517ff32b65a00bf9c95fd96205fa24f40f196149fbcb8e7: Status 404 returned error can't find the container with id 703babb3dc0a96fd4517ff32b65a00bf9c95fd96205fa24f40f196149fbcb8e7
Sep 10 21:36:08.472303 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:08.472275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q" event={"ID":"cf53bd83-4d28-438f-b690-a2a136147405","Type":"ContainerStarted","Data":"703babb3dc0a96fd4517ff32b65a00bf9c95fd96205fa24f40f196149fbcb8e7"}
Sep 10 21:36:09.478853 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:09.478817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q" event={"ID":"cf53bd83-4d28-438f-b690-a2a136147405","Type":"ContainerStarted","Data":"20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a"}
Sep 10 21:36:09.496837 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:09.496788    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q" podStartSLOduration=2.496771351 podStartE2EDuration="2.496771351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:36:09.494828144 +0000 UTC m=+1008.142742313" watchObservedRunningTime="2026-09-10 21:36:09.496771351 +0000 UTC m=+1008.144685540"
Sep 10 21:36:11.191652 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:11.191617    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"]
Sep 10 21:36:12.404636 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:12.404594    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"]
Sep 10 21:36:18.138073 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:18.138030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:20.408063 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:20.408020    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:36:26.508445 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.508368    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q" podUID="cf53bd83-4d28-438f-b690-a2a136147405" containerName="main" containerID="cri-o://20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a" gracePeriod=45
Sep 10 21:36:26.751010 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.750985    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:26.827200 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.827115    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-model-cache\") pod \"cf53bd83-4d28-438f-b690-a2a136147405\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") "
Sep 10 21:36:26.827200 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.827154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-tmp-dir\") pod \"cf53bd83-4d28-438f-b690-a2a136147405\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") "
Sep 10 21:36:26.827394 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.827232    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-home\") pod \"cf53bd83-4d28-438f-b690-a2a136147405\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") "
Sep 10 21:36:26.827394 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.827255    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-dshm\") pod \"cf53bd83-4d28-438f-b690-a2a136147405\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") "
Sep 10 21:36:26.827394 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.827275    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4srtz\" (UniqueName: \"kubernetes.io/projected/cf53bd83-4d28-438f-b690-a2a136147405-kube-api-access-4srtz\") pod \"cf53bd83-4d28-438f-b690-a2a136147405\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") "
Sep 10 21:36:26.827394 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.827299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf53bd83-4d28-438f-b690-a2a136147405-tls-certs\") pod \"cf53bd83-4d28-438f-b690-a2a136147405\" (UID: \"cf53bd83-4d28-438f-b690-a2a136147405\") "
Sep 10 21:36:26.827707 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.827487    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf53bd83-4d28-438f-b690-a2a136147405" (UID: "cf53bd83-4d28-438f-b690-a2a136147405"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:36:26.827707 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.827505    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf53bd83-4d28-438f-b690-a2a136147405" (UID: "cf53bd83-4d28-438f-b690-a2a136147405"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:36:26.827707 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.827544    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-home" (OuterVolumeSpecName: "home") pod "cf53bd83-4d28-438f-b690-a2a136147405" (UID: "cf53bd83-4d28-438f-b690-a2a136147405"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:36:26.829751 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.829718    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf53bd83-4d28-438f-b690-a2a136147405-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf53bd83-4d28-438f-b690-a2a136147405" (UID: "cf53bd83-4d28-438f-b690-a2a136147405"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:36:26.829878 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.829792    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf53bd83-4d28-438f-b690-a2a136147405-kube-api-access-4srtz" (OuterVolumeSpecName: "kube-api-access-4srtz") pod "cf53bd83-4d28-438f-b690-a2a136147405" (UID: "cf53bd83-4d28-438f-b690-a2a136147405"). InnerVolumeSpecName "kube-api-access-4srtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:36:26.829878 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.829824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-dshm" (OuterVolumeSpecName: "dshm") pod "cf53bd83-4d28-438f-b690-a2a136147405" (UID: "cf53bd83-4d28-438f-b690-a2a136147405"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:36:26.928162 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.928118    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:26.928162 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.928157    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:26.928162 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.928169    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4srtz\" (UniqueName: \"kubernetes.io/projected/cf53bd83-4d28-438f-b690-a2a136147405-kube-api-access-4srtz\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:26.928401 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.928179    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf53bd83-4d28-438f-b690-a2a136147405-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:26.928401 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.928188    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:26.928401 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:26.928196    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf53bd83-4d28-438f-b690-a2a136147405-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:27.424753 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.424693    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj" podUID="5c985a09-4f60-4d6a-a4ac-3a25f4432efc" containerName="main" containerID="cri-o://cd0facaf77ec9edbf2db62a50aa30aaa14967a3804fd768f3ca6dfeddb7d185c" gracePeriod=45
Sep 10 21:36:27.580278 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.580240    2580 generic.go:356] "Generic (PLEG): container finished" podID="5c985a09-4f60-4d6a-a4ac-3a25f4432efc" containerID="cd0facaf77ec9edbf2db62a50aa30aaa14967a3804fd768f3ca6dfeddb7d185c" exitCode=0
Sep 10 21:36:27.580724 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.580322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj" event={"ID":"5c985a09-4f60-4d6a-a4ac-3a25f4432efc","Type":"ContainerDied","Data":"cd0facaf77ec9edbf2db62a50aa30aaa14967a3804fd768f3ca6dfeddb7d185c"}
Sep 10 21:36:27.581837 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.581806    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf53bd83-4d28-438f-b690-a2a136147405" containerID="20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a" exitCode=0
Sep 10 21:36:27.581985 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.581841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q" event={"ID":"cf53bd83-4d28-438f-b690-a2a136147405","Type":"ContainerDied","Data":"20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a"}
Sep 10 21:36:27.581985 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.581870    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"
Sep 10 21:36:27.581985 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.581889    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q" event={"ID":"cf53bd83-4d28-438f-b690-a2a136147405","Type":"ContainerDied","Data":"703babb3dc0a96fd4517ff32b65a00bf9c95fd96205fa24f40f196149fbcb8e7"}
Sep 10 21:36:27.581985 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.581913    2580 scope.go:117] "RemoveContainer" containerID="20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a"
Sep 10 21:36:27.591653 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.591635    2580 scope.go:117] "RemoveContainer" containerID="20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a"
Sep 10 21:36:27.591924 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:36:27.591905    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a\": container with ID starting with 20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a not found: ID does not exist" containerID="20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a"
Sep 10 21:36:27.591977 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.591932    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a"} err="failed to get container status \"20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a\": rpc error: code = NotFound desc = could not find container \"20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a\": container with ID starting with 20c7eb6d8f07ddeacd43bb2cd6c45c5985fd488cd72e1db3deada9967992635a not found: ID does not exist"
Sep 10 21:36:27.602258 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.602186    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"]
Sep 10 21:36:27.606476 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.606451    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-295803/noten-model-295803-kserve-7c47b99475-58f8q"]
Sep 10 21:36:27.684447 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.684353    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:36:27.735317 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735284    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-model-cache\") pod \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") "
Sep 10 21:36:27.735511 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-home\") pod \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") "
Sep 10 21:36:27.735511 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbngc\" (UniqueName: \"kubernetes.io/projected/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-kube-api-access-hbngc\") pod \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") "
Sep 10 21:36:27.735511 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735485    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tmp-dir\") pod \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") "
Sep 10 21:36:27.735511 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tls-certs\") pod \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") "
Sep 10 21:36:27.735728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735523    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-dshm\") pod \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\" (UID: \"5c985a09-4f60-4d6a-a4ac-3a25f4432efc\") "
Sep 10 21:36:27.735728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c985a09-4f60-4d6a-a4ac-3a25f4432efc" (UID: "5c985a09-4f60-4d6a-a4ac-3a25f4432efc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:36:27.735821 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735753    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-home" (OuterVolumeSpecName: "home") pod "5c985a09-4f60-4d6a-a4ac-3a25f4432efc" (UID: "5c985a09-4f60-4d6a-a4ac-3a25f4432efc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:36:27.735821 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735769    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c985a09-4f60-4d6a-a4ac-3a25f4432efc" (UID: "5c985a09-4f60-4d6a-a4ac-3a25f4432efc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:36:27.735892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735853    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:27.735892 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735884    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:27.735961 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.735898    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:27.737937 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.737909    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-dshm" (OuterVolumeSpecName: "dshm") pod "5c985a09-4f60-4d6a-a4ac-3a25f4432efc" (UID: "5c985a09-4f60-4d6a-a4ac-3a25f4432efc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:36:27.738063 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.737940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-kube-api-access-hbngc" (OuterVolumeSpecName: "kube-api-access-hbngc") pod "5c985a09-4f60-4d6a-a4ac-3a25f4432efc" (UID: "5c985a09-4f60-4d6a-a4ac-3a25f4432efc"). InnerVolumeSpecName "kube-api-access-hbngc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:36:27.738063 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.738005    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c985a09-4f60-4d6a-a4ac-3a25f4432efc" (UID: "5c985a09-4f60-4d6a-a4ac-3a25f4432efc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:36:27.820755 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.820720    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf53bd83-4d28-438f-b690-a2a136147405" path="/var/lib/kubelet/pods/cf53bd83-4d28-438f-b690-a2a136147405/volumes"
Sep 10 21:36:27.836951 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.836927    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbngc\" (UniqueName: \"kubernetes.io/projected/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-kube-api-access-hbngc\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:27.837053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.836954    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:27.837053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:27.836968    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c985a09-4f60-4d6a-a4ac-3a25f4432efc-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:28.589324 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:28.589291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj" event={"ID":"5c985a09-4f60-4d6a-a4ac-3a25f4432efc","Type":"ContainerDied","Data":"05b294174a728f91562b5f2ad58512eda9addc6b16ac499c85b2d9e36ddfa86e"}
Sep 10 21:36:28.589795 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:28.589327    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"
Sep 10 21:36:28.589795 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:28.589340    2580 scope.go:117] "RemoveContainer" containerID="cd0facaf77ec9edbf2db62a50aa30aaa14967a3804fd768f3ca6dfeddb7d185c"
Sep 10 21:36:28.604581 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:28.604555    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"]
Sep 10 21:36:28.607997 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:28.607971    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5bba43/test-model-5bba43-kserve-76cc567b7f-q48vj"]
Sep 10 21:36:29.822567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:29.822531    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c985a09-4f60-4d6a-a4ac-3a25f4432efc" path="/var/lib/kubelet/pods/5c985a09-4f60-4d6a-a4ac-3a25f4432efc/volumes"
Sep 10 21:36:30.505740 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:30.505700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:36:40.833159 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:40.833128    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"]
Sep 10 21:36:40.833627 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:40.833359    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct" podUID="71f6f6db-e60b-42c1-a6c4-43c7cc310bc8" containerName="maas-api" containerID="cri-o://d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca" gracePeriod=30
Sep 10 21:36:40.894965 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:40.894926    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"]
Sep 10 21:36:40.895257 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:40.895197    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl" podUID="00297454-d8b5-422c-82b0-abdfbad07950" containerName="payload-processing" containerID="cri-o://969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534" gracePeriod=30
Sep 10 21:36:40.902645 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:40.902610    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"]
Sep 10 21:36:40.902877 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:40.902852    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48" podUID="1d7617ce-25db-48a7-b2c1-9e86f3cf9c40" containerName="payload-pre-processing" containerID="cri-o://43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e" gracePeriod=30
Sep 10 21:36:41.184713 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.184679    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:36:41.188542 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.188520    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:36:41.252946 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.252920    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00297454-d8b5-422c-82b0-abdfbad07950-plugins-config-volume\") pod \"00297454-d8b5-422c-82b0-abdfbad07950\" (UID: \"00297454-d8b5-422c-82b0-abdfbad07950\") "
Sep 10 21:36:41.253108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.253000    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzhwh\" (UniqueName: \"kubernetes.io/projected/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-kube-api-access-rzhwh\") pod \"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40\" (UID: \"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40\") "
Sep 10 21:36:41.253108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.253021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-plugins-config-volume\") pod \"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40\" (UID: \"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40\") "
Sep 10 21:36:41.253108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.253078    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjfqd\" (UniqueName: \"kubernetes.io/projected/00297454-d8b5-422c-82b0-abdfbad07950-kube-api-access-jjfqd\") pod \"00297454-d8b5-422c-82b0-abdfbad07950\" (UID: \"00297454-d8b5-422c-82b0-abdfbad07950\") "
Sep 10 21:36:41.253373 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.253343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00297454-d8b5-422c-82b0-abdfbad07950-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00297454-d8b5-422c-82b0-abdfbad07950" (UID: "00297454-d8b5-422c-82b0-abdfbad07950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:36:41.253504 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.253371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d7617ce-25db-48a7-b2c1-9e86f3cf9c40" (UID: "1d7617ce-25db-48a7-b2c1-9e86f3cf9c40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:36:41.256368 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.256340    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-kube-api-access-rzhwh" (OuterVolumeSpecName: "kube-api-access-rzhwh") pod "1d7617ce-25db-48a7-b2c1-9e86f3cf9c40" (UID: "1d7617ce-25db-48a7-b2c1-9e86f3cf9c40"). InnerVolumeSpecName "kube-api-access-rzhwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:36:41.256529 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.256494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00297454-d8b5-422c-82b0-abdfbad07950-kube-api-access-jjfqd" (OuterVolumeSpecName: "kube-api-access-jjfqd") pod "00297454-d8b5-422c-82b0-abdfbad07950" (UID: "00297454-d8b5-422c-82b0-abdfbad07950"). InnerVolumeSpecName "kube-api-access-jjfqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:36:41.274554 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.274533    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:36:41.353786 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.353750    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwv9f\" (UniqueName: \"kubernetes.io/projected/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-kube-api-access-bwv9f\") pod \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") "
Sep 10 21:36:41.353964 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.353833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-openshift-service-ca\") pod \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") "
Sep 10 21:36:41.353964 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.353861    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-metrics-certs\") pod \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") "
Sep 10 21:36:41.353964 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.353924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-maas-api-tls\") pod \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\" (UID: \"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8\") "
Sep 10 21:36:41.354206 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.354191    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjfqd\" (UniqueName: \"kubernetes.io/projected/00297454-d8b5-422c-82b0-abdfbad07950-kube-api-access-jjfqd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:41.354273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.354195    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71f6f6db-e60b-42c1-a6c4-43c7cc310bc8" (UID: "71f6f6db-e60b-42c1-a6c4-43c7cc310bc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:36:41.354273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.354212    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00297454-d8b5-422c-82b0-abdfbad07950-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:41.354273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.354226    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzhwh\" (UniqueName: \"kubernetes.io/projected/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-kube-api-access-rzhwh\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:41.354273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.354241    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:41.355948 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.355900    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-kube-api-access-bwv9f" (OuterVolumeSpecName: "kube-api-access-bwv9f") pod "71f6f6db-e60b-42c1-a6c4-43c7cc310bc8" (UID: "71f6f6db-e60b-42c1-a6c4-43c7cc310bc8"). InnerVolumeSpecName "kube-api-access-bwv9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:36:41.356012 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.355982    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71f6f6db-e60b-42c1-a6c4-43c7cc310bc8" (UID: "71f6f6db-e60b-42c1-a6c4-43c7cc310bc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:36:41.356045 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.356007    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71f6f6db-e60b-42c1-a6c4-43c7cc310bc8" (UID: "71f6f6db-e60b-42c1-a6c4-43c7cc310bc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:36:41.455592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.455555    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-openshift-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:41.455592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.455586    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-metrics-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:41.455592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.455597    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-maas-api-tls\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:41.455822 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.455605    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwv9f\" (UniqueName: \"kubernetes.io/projected/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8-kube-api-access-bwv9f\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:41.664897 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.664811    2580 generic.go:356] "Generic (PLEG): container finished" podID="71f6f6db-e60b-42c1-a6c4-43c7cc310bc8" containerID="d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca" exitCode=0
Sep 10 21:36:41.664897 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.664855    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"
Sep 10 21:36:41.665105 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.664894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct" event={"ID":"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8","Type":"ContainerDied","Data":"d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca"}
Sep 10 21:36:41.665105 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.664963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct" event={"ID":"71f6f6db-e60b-42c1-a6c4-43c7cc310bc8","Type":"ContainerDied","Data":"72ab462e65dafcef46bc6b27c30e83c53589ef9c7f7bf52ba2de2390803d921f"}
Sep 10 21:36:41.665105 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.664996    2580 scope.go:117] "RemoveContainer" containerID="d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca"
Sep 10 21:36:41.666262 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.666241    2580 generic.go:356] "Generic (PLEG): container finished" podID="00297454-d8b5-422c-82b0-abdfbad07950" containerID="969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534" exitCode=0
Sep 10 21:36:41.666335 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.666291    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"
Sep 10 21:36:41.666398 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.666329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl" event={"ID":"00297454-d8b5-422c-82b0-abdfbad07950","Type":"ContainerDied","Data":"969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534"}
Sep 10 21:36:41.666398 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.666387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl" event={"ID":"00297454-d8b5-422c-82b0-abdfbad07950","Type":"ContainerDied","Data":"9dd839e7e64b885794d7b929c8357d991972e31e9b3dc153a96a17e2001a7b09"}
Sep 10 21:36:41.667824 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.667802    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d7617ce-25db-48a7-b2c1-9e86f3cf9c40" containerID="43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e" exitCode=0
Sep 10 21:36:41.667949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.667843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48" event={"ID":"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40","Type":"ContainerDied","Data":"43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e"}
Sep 10 21:36:41.667949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.667849    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"
Sep 10 21:36:41.667949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.667876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48" event={"ID":"1d7617ce-25db-48a7-b2c1-9e86f3cf9c40","Type":"ContainerDied","Data":"a6c0f39133b2b5da84d5dcd999ad71a3179c112def895361ecfff7f633263bfb"}
Sep 10 21:36:41.676465 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.676406    2580 scope.go:117] "RemoveContainer" containerID="d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca"
Sep 10 21:36:41.676767 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:36:41.676739    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca\": container with ID starting with d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca not found: ID does not exist" containerID="d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca"
Sep 10 21:36:41.676819 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.676771    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca"} err="failed to get container status \"d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca\": rpc error: code = NotFound desc = could not find container \"d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca\": container with ID starting with d8549ab86a71c5d30aacfc26d5087c798a0e32de0582b014254a1a69f4ae68ca not found: ID does not exist"
Sep 10 21:36:41.676819 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.676792    2580 scope.go:117] "RemoveContainer" containerID="969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534"
Sep 10 21:36:41.684582 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.684528    2580 scope.go:117] "RemoveContainer" containerID="969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534"
Sep 10 21:36:41.684806 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:36:41.684786    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534\": container with ID starting with 969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534 not found: ID does not exist" containerID="969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534"
Sep 10 21:36:41.684860 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.684812    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534"} err="failed to get container status \"969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534\": rpc error: code = NotFound desc = could not find container \"969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534\": container with ID starting with 969770fcfd7474c0fae951d5c48893e94f6294fd7cc258adbe8dd303bf8a1534 not found: ID does not exist"
Sep 10 21:36:41.684860 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.684828    2580 scope.go:117] "RemoveContainer" containerID="43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e"
Sep 10 21:36:41.687622 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.687604    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"]
Sep 10 21:36:41.693010 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.692993    2580 scope.go:117] "RemoveContainer" containerID="43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e"
Sep 10 21:36:41.693287 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:36:41.693271    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e\": container with ID starting with 43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e not found: ID does not exist" containerID="43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e"
Sep 10 21:36:41.693336 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.693294    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e"} err="failed to get container status \"43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e\": rpc error: code = NotFound desc = could not find container \"43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e\": container with ID starting with 43836c7db310c9f5a6dc006272aab938b48d0c9f15e5068f3caa842d241f9e4e not found: ID does not exist"
Sep 10 21:36:41.694205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.694187    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bba43-78c7665787-nzkct"]
Sep 10 21:36:41.704883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.704864    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"]
Sep 10 21:36:41.709100 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.709080    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bba43-5f88579f97-wz8jl"]
Sep 10 21:36:41.720181 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.720162    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"]
Sep 10 21:36:41.724506 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.724488    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5bba43-57798f879d-99k48"]
Sep 10 21:36:41.820289 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.820258    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00297454-d8b5-422c-82b0-abdfbad07950" path="/var/lib/kubelet/pods/00297454-d8b5-422c-82b0-abdfbad07950/volumes"
Sep 10 21:36:41.820646 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.820632    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d7617ce-25db-48a7-b2c1-9e86f3cf9c40" path="/var/lib/kubelet/pods/1d7617ce-25db-48a7-b2c1-9e86f3cf9c40/volumes"
Sep 10 21:36:41.820964 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:41.820953    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f6f6db-e60b-42c1-a6c4-43c7cc310bc8" path="/var/lib/kubelet/pods/71f6f6db-e60b-42c1-a6c4-43c7cc310bc8/volumes"
Sep 10 21:36:43.805756 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:43.805720    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:36:48.578739 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.578707    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"]
Sep 10 21:36:48.579108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579052    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7617ce-25db-48a7-b2c1-9e86f3cf9c40" containerName="payload-pre-processing"
Sep 10 21:36:48.579108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579063    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7617ce-25db-48a7-b2c1-9e86f3cf9c40" containerName="payload-pre-processing"
Sep 10 21:36:48.579108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579075    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf53bd83-4d28-438f-b690-a2a136147405" containerName="main"
Sep 10 21:36:48.579108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579081    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf53bd83-4d28-438f-b690-a2a136147405" containerName="main"
Sep 10 21:36:48.579108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579089    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f6f6db-e60b-42c1-a6c4-43c7cc310bc8" containerName="maas-api"
Sep 10 21:36:48.579108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579094    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f6f6db-e60b-42c1-a6c4-43c7cc310bc8" containerName="maas-api"
Sep 10 21:36:48.579108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579100    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00297454-d8b5-422c-82b0-abdfbad07950" containerName="payload-processing"
Sep 10 21:36:48.579108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579106    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="00297454-d8b5-422c-82b0-abdfbad07950" containerName="payload-processing"
Sep 10 21:36:48.579367 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579120    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c985a09-4f60-4d6a-a4ac-3a25f4432efc" containerName="main"
Sep 10 21:36:48.579367 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579125    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c985a09-4f60-4d6a-a4ac-3a25f4432efc" containerName="main"
Sep 10 21:36:48.579367 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579174    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f6f6db-e60b-42c1-a6c4-43c7cc310bc8" containerName="maas-api"
Sep 10 21:36:48.579367 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579182    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="00297454-d8b5-422c-82b0-abdfbad07950" containerName="payload-processing"
Sep 10 21:36:48.579367 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579188    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c985a09-4f60-4d6a-a4ac-3a25f4432efc" containerName="main"
Sep 10 21:36:48.579367 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579193    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf53bd83-4d28-438f-b690-a2a136147405" containerName="main"
Sep 10 21:36:48.579367 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.579201    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7617ce-25db-48a7-b2c1-9e86f3cf9c40" containerName="payload-pre-processing"
Sep 10 21:36:48.582085 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.582066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.586042 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.586017    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:36:48.586776 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.586761    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-2d9e68-openshift-default-dockercfg-l48n8\""
Sep 10 21:36:48.603684 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.603659    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"]
Sep 10 21:36:48.722390 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.722356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-socket\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.722570 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.722396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-podinfo\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.722570 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.722483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-credential-socket\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.722570 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.722536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-token\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.722570 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.722559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-envoy\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.722715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.722582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.722715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.722620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqpbq\" (UniqueName: \"kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-kube-api-access-kqpbq\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.722715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.722660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-data\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.722816 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.722708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-certs\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.823388 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.823358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-token\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.823558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.823391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-envoy\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.823558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.823437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.823558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.823452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpbq\" (UniqueName: \"kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-kube-api-access-kqpbq\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.823558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.823480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-data\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.823558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.823512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-certs\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.823558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.823542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-socket\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.823861 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.823576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-podinfo\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.823861 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.823624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-credential-socket\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.823970 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.823885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-data\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.824025 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.823980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-certs\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.824025 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.824016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-socket\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.824170 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.824147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-credential-socket\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.824288 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.824268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.825919 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.825889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-envoy\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.826269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.826252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-podinfo\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.835641 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.835588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-token\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.835749 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.835732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpbq\" (UniqueName: \"kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-kube-api-access-kqpbq\") pod \"e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") " pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:48.891867 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:48.891838    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:49.194077 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:49.194049    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"]
Sep 10 21:36:49.196762 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:36:49.196736    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfb0d90_3326_41a7_b8c0_4b8d5bfb6d3b.slice/crio-7ef1f4f6f0bcd7e5e033e9d29cc097f7e0f19faed21fafb5499223babf2eabc2 WatchSource:0}: Error finding container 7ef1f4f6f0bcd7e5e033e9d29cc097f7e0f19faed21fafb5499223babf2eabc2: Status 404 returned error can't find the container with id 7ef1f4f6f0bcd7e5e033e9d29cc097f7e0f19faed21fafb5499223babf2eabc2
Sep 10 21:36:49.199098 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:49.199064    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:36:49.199204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:49.199140    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:36:49.199204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:49.199182    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:36:49.712585 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:49.712549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" event={"ID":"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b","Type":"ContainerStarted","Data":"da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680"}
Sep 10 21:36:49.713054 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:49.712648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" event={"ID":"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b","Type":"ContainerStarted","Data":"7ef1f4f6f0bcd7e5e033e9d29cc097f7e0f19faed21fafb5499223babf2eabc2"}
Sep 10 21:36:49.739549 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:49.739501    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" podStartSLOduration=1.739406319 podStartE2EDuration="1.739406319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:36:49.739048537 +0000 UTC m=+1048.386962706" watchObservedRunningTime="2026-09-10 21:36:49.739406319 +0000 UTC m=+1048.387320465"
Sep 10 21:36:49.892484 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:49.892447    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:50.897803 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:50.897771    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:51.416707 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:51.416676    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"]
Sep 10 21:36:51.419832 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:51.419817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"
Sep 10 21:36:51.422081 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:51.422058    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wg7l\""
Sep 10 21:36:51.422201 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:51.422101    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:36:51.428795 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:51.428770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"]
Sep 10 21:36:51.550736 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:51.550700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e2ddf-b517-4cde-810c-94da2204836d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb\" (UID: \"9e5e2ddf-b517-4cde-810c-94da2204836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"
Sep 10 21:36:51.651861 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:51.651823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e2ddf-b517-4cde-810c-94da2204836d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb\" (UID: \"9e5e2ddf-b517-4cde-810c-94da2204836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"
Sep 10 21:36:51.652467 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:51.652446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e2ddf-b517-4cde-810c-94da2204836d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb\" (UID: \"9e5e2ddf-b517-4cde-810c-94da2204836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"
Sep 10 21:36:51.723570 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:51.723535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:51.724461 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:51.724439    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:36:51.731812 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:51.731795    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"
Sep 10 21:36:52.020711 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:52.020688    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"]
Sep 10 21:36:52.023190 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:36:52.023158    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5e2ddf_b517_4cde_810c_94da2204836d.slice/crio-736fbf0bb6f5fc4c5c89700f1dcdab03c986130fed1ae3436c98a8705497445e WatchSource:0}: Error finding container 736fbf0bb6f5fc4c5c89700f1dcdab03c986130fed1ae3436c98a8705497445e: Status 404 returned error can't find the container with id 736fbf0bb6f5fc4c5c89700f1dcdab03c986130fed1ae3436c98a8705497445e
Sep 10 21:36:52.729715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:52.729664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb" event={"ID":"9e5e2ddf-b517-4cde-810c-94da2204836d","Type":"ContainerStarted","Data":"dcdecf942bba27c3df0d9925a5d4ac0043d06e56a2013a83ecf35512edb8933a"}
Sep 10 21:36:52.729876 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:52.729732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb" event={"ID":"9e5e2ddf-b517-4cde-810c-94da2204836d","Type":"ContainerStarted","Data":"736fbf0bb6f5fc4c5c89700f1dcdab03c986130fed1ae3436c98a8705497445e"}
Sep 10 21:36:52.744517 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:52.744454    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb" podStartSLOduration=1.744434548 podStartE2EDuration="1.744434548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:36:52.743729541 +0000 UTC m=+1051.391643710" watchObservedRunningTime="2026-09-10 21:36:52.744434548 +0000 UTC m=+1051.392348711"
Sep 10 21:36:52.900798 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:52.900767    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"]
Sep 10 21:36:52.904064 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:52.904046    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:52.907552 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:52.907501    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-2d9e68-serving-cert\""
Sep 10 21:36:52.907552 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:52.907543    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 21:36:52.907764 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:52.907564    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nmksg\""
Sep 10 21:36:52.907943 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:52.907925    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 21:36:52.916000 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:52.915924    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"]
Sep 10 21:36:53.066229 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.066145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.066229 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.066198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25316051-d336-49b0-b1c5-e997d8ec5d8b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.066697 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.066296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rzc\" (UniqueName: \"kubernetes.io/projected/25316051-d336-49b0-b1c5-e997d8ec5d8b-kube-api-access-x8rzc\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.066697 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.066348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.166995 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.166958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.167184 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.167014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25316051-d336-49b0-b1c5-e997d8ec5d8b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.167184 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.167068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rzc\" (UniqueName: \"kubernetes.io/projected/25316051-d336-49b0-b1c5-e997d8ec5d8b-kube-api-access-x8rzc\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.167184 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.167094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.167184 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:36:53.167141    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-serving-cert: secret "maas-api-e2e-shared-a-w3-2d9e68-serving-cert" not found
Sep 10 21:36:53.167378 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:36:53.167218    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-maas-api-tls podName:25316051-d336-49b0-b1c5-e997d8ec5d8b nodeName:}" failed. No retries permitted until 2026-09-10 21:36:53.667192923 +0000 UTC m=+1052.315107086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-maas-api-tls") pod "maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f" (UID: "25316051-d336-49b0-b1c5-e997d8ec5d8b") : secret "maas-api-e2e-shared-a-w3-2d9e68-serving-cert" not found
Sep 10 21:36:53.167817 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.167800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25316051-d336-49b0-b1c5-e997d8ec5d8b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.169680 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.169648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.176982 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.176959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rzc\" (UniqueName: \"kubernetes.io/projected/25316051-d336-49b0-b1c5-e997d8ec5d8b-kube-api-access-x8rzc\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.521161 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.521121    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"]
Sep 10 21:36:53.524617 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.524595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:36:53.526998 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.526981    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2d9e68\""
Sep 10 21:36:53.527116 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.527095    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2d9e68-dockercfg-mpds9\""
Sep 10 21:36:53.535337 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.535313    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"]
Sep 10 21:36:53.678969 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.678928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.679179 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.679067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slxw\" (UniqueName: \"kubernetes.io/projected/26e0dd97-a253-48c6-965d-70921a610bce-kube-api-access-7slxw\") pod \"payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b\" (UID: \"26e0dd97-a253-48c6-965d-70921a610bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:36:53.679179 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.679113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e0dd97-a253-48c6-965d-70921a610bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b\" (UID: \"26e0dd97-a253-48c6-965d-70921a610bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:36:53.681755 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.681725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.736334 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.736301    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e5e2ddf-b517-4cde-810c-94da2204836d" containerID="dcdecf942bba27c3df0d9925a5d4ac0043d06e56a2013a83ecf35512edb8933a" exitCode=0
Sep 10 21:36:53.736492 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.736387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb" event={"ID":"9e5e2ddf-b517-4cde-810c-94da2204836d","Type":"ContainerDied","Data":"dcdecf942bba27c3df0d9925a5d4ac0043d06e56a2013a83ecf35512edb8933a"}
Sep 10 21:36:53.779947 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.779844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slxw\" (UniqueName: \"kubernetes.io/projected/26e0dd97-a253-48c6-965d-70921a610bce-kube-api-access-7slxw\") pod \"payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b\" (UID: \"26e0dd97-a253-48c6-965d-70921a610bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:36:53.779947 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.779888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e0dd97-a253-48c6-965d-70921a610bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b\" (UID: \"26e0dd97-a253-48c6-965d-70921a610bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:36:53.780634 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.780600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e0dd97-a253-48c6-965d-70921a610bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b\" (UID: \"26e0dd97-a253-48c6-965d-70921a610bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:36:53.787757 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.787729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slxw\" (UniqueName: \"kubernetes.io/projected/26e0dd97-a253-48c6-965d-70921a610bce-kube-api-access-7slxw\") pod \"payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b\" (UID: \"26e0dd97-a253-48c6-965d-70921a610bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:36:53.816979 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.816944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:53.835969 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:53.835947    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:36:54.094974 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.094938    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"]
Sep 10 21:36:54.097238 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:36:54.097210    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25316051_d336_49b0_b1c5_e997d8ec5d8b.slice/crio-07b6916e2d5e32a533510aa97bccc4f6b7bafac0284c23adee61033e84bd6d26 WatchSource:0}: Error finding container 07b6916e2d5e32a533510aa97bccc4f6b7bafac0284c23adee61033e84bd6d26: Status 404 returned error can't find the container with id 07b6916e2d5e32a533510aa97bccc4f6b7bafac0284c23adee61033e84bd6d26
Sep 10 21:36:54.098713 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.098689    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:36:54.113234 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.113168    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"]
Sep 10 21:36:54.115353 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:36:54.115325    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e0dd97_a253_48c6_965d_70921a610bce.slice/crio-c51abada3db7e3a7d172d0eb9d4fdac6e3038ca98c2d786e118538d2c4b4e246 WatchSource:0}: Error finding container c51abada3db7e3a7d172d0eb9d4fdac6e3038ca98c2d786e118538d2c4b4e246: Status 404 returned error can't find the container with id c51abada3db7e3a7d172d0eb9d4fdac6e3038ca98c2d786e118538d2c4b4e246
Sep 10 21:36:54.742804 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.742768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f" event={"ID":"25316051-d336-49b0-b1c5-e997d8ec5d8b","Type":"ContainerStarted","Data":"8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19"}
Sep 10 21:36:54.742991 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.742825    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f" event={"ID":"25316051-d336-49b0-b1c5-e997d8ec5d8b","Type":"ContainerStarted","Data":"07b6916e2d5e32a533510aa97bccc4f6b7bafac0284c23adee61033e84bd6d26"}
Sep 10 21:36:54.742991 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.742853    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:36:54.744445 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.744403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b" event={"ID":"26e0dd97-a253-48c6-965d-70921a610bce","Type":"ContainerStarted","Data":"c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7"}
Sep 10 21:36:54.744552 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.744461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b" event={"ID":"26e0dd97-a253-48c6-965d-70921a610bce","Type":"ContainerStarted","Data":"c51abada3db7e3a7d172d0eb9d4fdac6e3038ca98c2d786e118538d2c4b4e246"}
Sep 10 21:36:54.744552 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.744487    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:36:54.765168 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.765094    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f" podStartSLOduration=2.319048281 podStartE2EDuration="2.76505315s" totalImagesPullingTime="446.004869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:36:54.762577394 +0000 UTC m=+1053.410491564" watchObservedRunningTime="2026-09-10 21:36:54.76505315 +0000 UTC m=+1053.412967319"
Sep 10 21:36:54.781708 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.781651    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b" podStartSLOduration=1.781629962 podStartE2EDuration="1.781629962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:36:54.779095035 +0000 UTC m=+1053.427009215" watchObservedRunningTime="2026-09-10 21:36:54.781629962 +0000 UTC m=+1053.429544134"
Sep 10 21:36:54.881353 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.881328    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"
Sep 10 21:36:54.992998 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.992969    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e2ddf-b517-4cde-810c-94da2204836d-maas-api-service-ca\") pod \"9e5e2ddf-b517-4cde-810c-94da2204836d\" (UID: \"9e5e2ddf-b517-4cde-810c-94da2204836d\") "
Sep 10 21:36:54.993372 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:54.993350    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5e2ddf-b517-4cde-810c-94da2204836d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e5e2ddf-b517-4cde-810c-94da2204836d" (UID: "9e5e2ddf-b517-4cde-810c-94da2204836d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:36:55.094399 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:55.094316    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e2ddf-b517-4cde-810c-94da2204836d-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:36:55.751553 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:55.751517    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"
Sep 10 21:36:55.752025 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:55.751547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb" event={"ID":"9e5e2ddf-b517-4cde-810c-94da2204836d","Type":"ContainerDied","Data":"736fbf0bb6f5fc4c5c89700f1dcdab03c986130fed1ae3436c98a8705497445e"}
Sep 10 21:36:55.752025 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:55.751596    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="736fbf0bb6f5fc4c5c89700f1dcdab03c986130fed1ae3436c98a8705497445e"
Sep 10 21:36:56.306062 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:36:56.306022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:37:00.758366 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:00.758322    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:37:02.364861 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:02.364823    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"]
Sep 10 21:37:02.367062 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:02.367020    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c5c624-hc2qb"]
Sep 10 21:37:03.827123 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:03.827040    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e5e2ddf-b517-4cde-810c-94da2204836d" path="/var/lib/kubelet/pods/9e5e2ddf-b517-4cde-810c-94da2204836d/volumes"
Sep 10 21:37:05.754577 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:05.754544    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:37:06.705741 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:06.705708    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:37:06.725685 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:06.725648    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 10 21:37:06.725839 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:06.725711    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 10 21:37:06.817639 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:06.817552    2580 generic.go:356] "Generic (PLEG): container finished" podID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerID="da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680" exitCode=0
Sep 10 21:37:06.818005 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:06.817634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" event={"ID":"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b","Type":"ContainerDied","Data":"da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680"}
Sep 10 21:37:06.818048 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:06.818013    2580 scope.go:117] "RemoveContainer" containerID="da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680"
Sep 10 21:37:06.819138 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:06.819112    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:37:06.819211 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:06.819178    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:37:06.819253 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:06.819216    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:37:07.827955 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:07.827918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" event={"ID":"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b","Type":"ContainerStarted","Data":"ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0"}
Sep 10 21:37:07.892912 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:07.892869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:37:07.893196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:07.893172    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 10 21:37:07.893251 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:07.893229    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 10 21:37:08.893268 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:08.893227    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 10 21:37:08.893684 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:08.893290    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 10 21:37:09.892747 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:09.892710    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 10 21:37:09.892909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:09.892769    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 10 21:37:10.892495 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:10.892462    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 10 21:37:10.892887 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:10.892515    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 10 21:37:11.893190 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:11.893146    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 10 21:37:11.893629 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:11.893215    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 10 21:37:12.892861 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:12.892767    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 10 21:37:12.893031 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:12.892845    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 10 21:37:13.896679 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:13.896650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:37:13.897111 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:13.896907    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:37:13.897484 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:13.897467    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:37:27.409811 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:27.409770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:37:28.319742 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.319705    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"]
Sep 10 21:37:28.320096 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.320083    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5e2ddf-b517-4cde-810c-94da2204836d" containerName="revoke-keys"
Sep 10 21:37:28.320142 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.320097    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5e2ddf-b517-4cde-810c-94da2204836d" containerName="revoke-keys"
Sep 10 21:37:28.320182 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.320168    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5e2ddf-b517-4cde-810c-94da2204836d" containerName="revoke-keys"
Sep 10 21:37:28.323171 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.323153    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:37:28.325372 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.325343    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4b45c2\""
Sep 10 21:37:28.325372 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.325384    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4b45c2-dockercfg-xzsth\""
Sep 10 21:37:28.333982 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.333954    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"]
Sep 10 21:37:28.356183 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.356151    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"]
Sep 10 21:37:28.358797 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.358772    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:37:28.371675 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.371646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"]
Sep 10 21:37:28.426193 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.426134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgfr\" (UniqueName: \"kubernetes.io/projected/69e231f2-2929-437a-be9c-134228a2719c-kube-api-access-qpgfr\") pod \"payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv\" (UID: \"69e231f2-2929-437a-be9c-134228a2719c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:37:28.426620 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.426214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e231f2-2929-437a-be9c-134228a2719c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv\" (UID: \"69e231f2-2929-437a-be9c-134228a2719c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:37:28.526921 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.526876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgfr\" (UniqueName: \"kubernetes.io/projected/69e231f2-2929-437a-be9c-134228a2719c-kube-api-access-qpgfr\") pod \"payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv\" (UID: \"69e231f2-2929-437a-be9c-134228a2719c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:37:28.526921 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.526934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw\" (UID: \"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:37:28.527210 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.526956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e231f2-2929-437a-be9c-134228a2719c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv\" (UID: \"69e231f2-2929-437a-be9c-134228a2719c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:37:28.527210 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.527002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mkd\" (UniqueName: \"kubernetes.io/projected/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-kube-api-access-95mkd\") pod \"payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw\" (UID: \"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:37:28.527717 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.527697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e231f2-2929-437a-be9c-134228a2719c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv\" (UID: \"69e231f2-2929-437a-be9c-134228a2719c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:37:28.534657 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.534631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgfr\" (UniqueName: \"kubernetes.io/projected/69e231f2-2929-437a-be9c-134228a2719c-kube-api-access-qpgfr\") pod \"payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv\" (UID: \"69e231f2-2929-437a-be9c-134228a2719c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:37:28.628194 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.628087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw\" (UID: \"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:37:28.628194 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.628182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mkd\" (UniqueName: \"kubernetes.io/projected/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-kube-api-access-95mkd\") pod \"payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw\" (UID: \"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:37:28.628930 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.628901    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw\" (UID: \"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:37:28.633992 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.633976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:37:28.636373 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.636301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mkd\" (UniqueName: \"kubernetes.io/projected/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-kube-api-access-95mkd\") pod \"payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw\" (UID: \"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:37:28.673268 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.673230    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:37:28.912131 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.912104    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"]
Sep 10 21:37:28.914124 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:37:28.914087    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e231f2_2929_437a_be9c_134228a2719c.slice/crio-a6ec4507693e5ce846fae15fa3a1e6b01f424f31fc9d4eeee5f4d132081720b4 WatchSource:0}: Error finding container a6ec4507693e5ce846fae15fa3a1e6b01f424f31fc9d4eeee5f4d132081720b4: Status 404 returned error can't find the container with id a6ec4507693e5ce846fae15fa3a1e6b01f424f31fc9d4eeee5f4d132081720b4
Sep 10 21:37:28.931960 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.931930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv" event={"ID":"69e231f2-2929-437a-be9c-134228a2719c","Type":"ContainerStarted","Data":"a6ec4507693e5ce846fae15fa3a1e6b01f424f31fc9d4eeee5f4d132081720b4"}
Sep 10 21:37:28.959103 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:28.959080    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"]
Sep 10 21:37:28.960942 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:37:28.960917    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6f83f9_f3c2_4cda_abdc_5bc7c2f69bf1.slice/crio-da30d86f43f103f1c36c6144a441fb21073dff5c56d1af2fbebeed673a100af2 WatchSource:0}: Error finding container da30d86f43f103f1c36c6144a441fb21073dff5c56d1af2fbebeed673a100af2: Status 404 returned error can't find the container with id da30d86f43f103f1c36c6144a441fb21073dff5c56d1af2fbebeed673a100af2
Sep 10 21:37:29.939499 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:29.939459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw" event={"ID":"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1","Type":"ContainerStarted","Data":"504e101b81165b5bb723dd0e6457b6027431be309200382b729a10415a1b310d"}
Sep 10 21:37:29.939993 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:29.939527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw" event={"ID":"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1","Type":"ContainerStarted","Data":"da30d86f43f103f1c36c6144a441fb21073dff5c56d1af2fbebeed673a100af2"}
Sep 10 21:37:29.939993 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:29.939560    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:37:29.941101 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:29.941070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv" event={"ID":"69e231f2-2929-437a-be9c-134228a2719c","Type":"ContainerStarted","Data":"8ad768c404609f1ac7ff9bb5c92ea9b43d4a362973e854452af34838a7682dcb"}
Sep 10 21:37:29.941205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:29.941115    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:37:29.958078 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:29.958022    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw" podStartSLOduration=1.95800031 podStartE2EDuration="1.95800031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:37:29.956605313 +0000 UTC m=+1088.604519472" watchObservedRunningTime="2026-09-10 21:37:29.95800031 +0000 UTC m=+1088.605914480"
Sep 10 21:37:29.971665 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:29.971600    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv" podStartSLOduration=1.9715791230000002 podStartE2EDuration="1.971579123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:37:29.971394553 +0000 UTC m=+1088.619308714" watchObservedRunningTime="2026-09-10 21:37:29.971579123 +0000 UTC m=+1088.619493296"
Sep 10 21:37:37.407923 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:37.407881    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:37:40.948477 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:40.948396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:37:40.948899 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:40.948537    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:37:51.105365 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.105331    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"]
Sep 10 21:37:51.107886 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.107867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.111128 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.110835    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d9e68\"/\"openshift-service-ca.crt\""
Sep 10 21:37:51.111128 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.110867    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d9e68\"/\"default-dockercfg-hrdn8\""
Sep 10 21:37:51.111128 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.110892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d9e68\"/\"auth-test-model-2d9e68-kserve-self-signed-certs\""
Sep 10 21:37:51.111128 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.110896    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d9e68\"/\"kube-root-ca.crt\""
Sep 10 21:37:51.119446 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.119333    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"]
Sep 10 21:37:51.237901 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.237866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-tmp-dir\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.238071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.237906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9453cde0-6173-43c3-861b-1738e2a6a4e0-tls-certs\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.238071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.237942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-dshm\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.238071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.237967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-home\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.238071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.237993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bth76\" (UniqueName: \"kubernetes.io/projected/9453cde0-6173-43c3-861b-1738e2a6a4e0-kube-api-access-bth76\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.238071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.238011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-model-cache\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.338887 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.338855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-tmp-dir\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.338887 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.338885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9453cde0-6173-43c3-861b-1738e2a6a4e0-tls-certs\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.339137 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.338908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-dshm\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.339137 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.338936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-home\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.339137 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.338959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bth76\" (UniqueName: \"kubernetes.io/projected/9453cde0-6173-43c3-861b-1738e2a6a4e0-kube-api-access-bth76\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.339137 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.338984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-model-cache\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.339337 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.339284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-tmp-dir\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.339407 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.339360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-model-cache\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.339483 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.339389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-home\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.341313 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.341290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-dshm\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.341576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.341561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9453cde0-6173-43c3-861b-1738e2a6a4e0-tls-certs\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.349699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.349677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bth76\" (UniqueName: \"kubernetes.io/projected/9453cde0-6173-43c3-861b-1738e2a6a4e0-kube-api-access-bth76\") pod \"auth-test-model-2d9e68-kserve-7866776f55-r6q4m\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.419493 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.419391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:37:51.690194 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:51.690168    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"]
Sep 10 21:37:51.692510 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:37:51.692475    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9453cde0_6173_43c3_861b_1738e2a6a4e0.slice/crio-e237b288a75882de2bf072ef98d405f5148c6fdf6e5475a8cd36ba86efd5a282 WatchSource:0}: Error finding container e237b288a75882de2bf072ef98d405f5148c6fdf6e5475a8cd36ba86efd5a282: Status 404 returned error can't find the container with id e237b288a75882de2bf072ef98d405f5148c6fdf6e5475a8cd36ba86efd5a282
Sep 10 21:37:52.054661 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.054561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m" event={"ID":"9453cde0-6173-43c3-861b-1738e2a6a4e0","Type":"ContainerStarted","Data":"e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f"}
Sep 10 21:37:52.054661 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.054614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m" event={"ID":"9453cde0-6173-43c3-861b-1738e2a6a4e0","Type":"ContainerStarted","Data":"e237b288a75882de2bf072ef98d405f5148c6fdf6e5475a8cd36ba86efd5a282"}
Sep 10 21:37:52.072492 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.072440    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m" podStartSLOduration=1.07239809 podStartE2EDuration="1.07239809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:37:52.071003953 +0000 UTC m=+1110.718918122" watchObservedRunningTime="2026-09-10 21:37:52.07239809 +0000 UTC m=+1110.720312257"
Sep 10 21:37:52.120793 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.120763    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"]
Sep 10 21:37:52.123532 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.123516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.125622 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.125599    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b4c9bb-openshift-default-dockercfg-kfdpz\""
Sep 10 21:37:52.136205 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.136177    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"]
Sep 10 21:37:52.248480 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.248447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqwfv\" (UniqueName: \"kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-kube-api-access-jqwfv\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.248647 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.248511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-socket\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.248647 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.248536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-credential-socket\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.248647 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.248556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-token\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.248647 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.248615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-podinfo\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.248786 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.248685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-certs\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.248786 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.248716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e0964b-d51f-44e3-83d4-546d9e7e7718-istiod-ca-cert\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.248786 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.248766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-envoy\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.248888 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.248789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-data\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.349829 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.349747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e0964b-d51f-44e3-83d4-546d9e7e7718-istiod-ca-cert\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.349829 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.349803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-envoy\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.349829 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.349824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-data\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.350083 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.349868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqwfv\" (UniqueName: \"kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-kube-api-access-jqwfv\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.350083 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.349900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-socket\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.350083 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.349928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-credential-socket\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.350083 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.349956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-token\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.350083 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.349984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-podinfo\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.350083 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.350024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-certs\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.350375 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.350266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-credential-socket\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.350375 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.350360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-socket\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.350515 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.350503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-data\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.350582 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.350564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e0964b-d51f-44e3-83d4-546d9e7e7718-istiod-ca-cert\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.350640 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.350615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-certs\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.352333 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.352314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-envoy\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.352783 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.352752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-podinfo\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.366279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.366251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqwfv\" (UniqueName: \"kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-kube-api-access-jqwfv\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.370197 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.370174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-token\") pod \"e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") " pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.433270 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.433244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:52.710130 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.710097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"]
Sep 10 21:37:52.713703 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:37:52.713673    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e0964b_d51f_44e3_83d4_546d9e7e7718.slice/crio-bf15faa13c6742391158658e08cc7a4dfcab6f06d623b17f7f2ed45719efccc0 WatchSource:0}: Error finding container bf15faa13c6742391158658e08cc7a4dfcab6f06d623b17f7f2ed45719efccc0: Status 404 returned error can't find the container with id bf15faa13c6742391158658e08cc7a4dfcab6f06d623b17f7f2ed45719efccc0
Sep 10 21:37:52.716017 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.715987    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:37:52.716094 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.716051    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:37:52.716094 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:52.716082    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:37:53.062333 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:53.062234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" event={"ID":"19e0964b-d51f-44e3-83d4-546d9e7e7718","Type":"ContainerStarted","Data":"1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264"}
Sep 10 21:37:53.062333 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:53.062297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" event={"ID":"19e0964b-d51f-44e3-83d4-546d9e7e7718","Type":"ContainerStarted","Data":"bf15faa13c6742391158658e08cc7a4dfcab6f06d623b17f7f2ed45719efccc0"}
Sep 10 21:37:53.082173 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:53.082123    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" podStartSLOduration=1.0821085990000001 podStartE2EDuration="1.082108599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:37:53.081263872 +0000 UTC m=+1111.729178081" watchObservedRunningTime="2026-09-10 21:37:53.082108599 +0000 UTC m=+1111.730022829"
Sep 10 21:37:53.433902 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:53.433807    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:53.439103 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:53.439075    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:54.068560 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:54.068529    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:54.069532 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:54.069508    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:37:55.919764 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:37:55.919724    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:38:01.419911 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:01.419871    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:38:01.419911 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:01.419922    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:38:01.432518 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:01.432490    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:38:02.123244 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:02.123207    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:38:09.424850 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:09.424812    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:38:14.174923 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:14.174885    2580 generic.go:356] "Generic (PLEG): container finished" podID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerID="1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264" exitCode=0
Sep 10 21:38:14.175364 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:14.174955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" event={"ID":"19e0964b-d51f-44e3-83d4-546d9e7e7718","Type":"ContainerDied","Data":"1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264"}
Sep 10 21:38:14.175364 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:14.175354    2580 scope.go:117] "RemoveContainer" containerID="1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264"
Sep 10 21:38:14.176441 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:14.176390    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:38:14.176550 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:14.176469    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:38:14.176550 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:14.176502    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:38:15.183589 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:15.183550    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" event={"ID":"19e0964b-d51f-44e3-83d4-546d9e7e7718","Type":"ContainerStarted","Data":"be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4"}
Sep 10 21:38:15.434123 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:15.434018    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:38:15.434264 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:15.434229    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 21:38:15.434320 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:15.434275    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 21:38:16.434477 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:16.434403    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 21:38:16.434882 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:16.434501    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 21:38:17.434118 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:17.434081    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 21:38:17.434312 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:17.434154    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 21:38:18.434060 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:18.434019    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 21:38:18.434527 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:18.434094    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 21:38:19.434434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:19.434370    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 21:38:19.434865 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:19.434462    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 21:38:20.437957 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:20.437920    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:38:20.438398 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:20.438199    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:38:20.438767 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:20.438750    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:38:36.650645 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.650613    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"]
Sep 10 21:38:36.653285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.653268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:38:36.655563 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.655540    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8c245620-dockercfg-fcdbh\""
Sep 10 21:38:36.655661 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.655611    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8c245620\""
Sep 10 21:38:36.663445 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.663400    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"]
Sep 10 21:38:36.676343 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.676316    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"]
Sep 10 21:38:36.678995 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.678980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:38:36.689860 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.689834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"]
Sep 10 21:38:36.796311 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.796267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcpll\" (UniqueName: \"kubernetes.io/projected/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-kube-api-access-kcpll\") pod \"payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8\" (UID: \"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9\") " pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:38:36.796510 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.796321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-plugins-config-volume\") pod \"payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8\" (UID: \"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9\") " pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:38:36.796510 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.796358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhjs\" (UniqueName: \"kubernetes.io/projected/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-kube-api-access-vdhjs\") pod \"payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg\" (UID: \"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:38:36.796510 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.796434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg\" (UID: \"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:38:36.897850 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.897803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpll\" (UniqueName: \"kubernetes.io/projected/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-kube-api-access-kcpll\") pod \"payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8\" (UID: \"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9\") " pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:38:36.898057 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.897876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-plugins-config-volume\") pod \"payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8\" (UID: \"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9\") " pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:38:36.898057 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.897916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhjs\" (UniqueName: \"kubernetes.io/projected/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-kube-api-access-vdhjs\") pod \"payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg\" (UID: \"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:38:36.898057 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.897974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg\" (UID: \"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:38:36.898740 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.898716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-plugins-config-volume\") pod \"payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8\" (UID: \"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9\") " pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:38:36.898815 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.898800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg\" (UID: \"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:38:36.906330 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.906272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhjs\" (UniqueName: \"kubernetes.io/projected/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-kube-api-access-vdhjs\") pod \"payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg\" (UID: \"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:38:36.906517 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.906494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpll\" (UniqueName: \"kubernetes.io/projected/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-kube-api-access-kcpll\") pod \"payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8\" (UID: \"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9\") " pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:38:36.963883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.963848    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:38:36.990965 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:36.990933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:38:37.237650 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:37.237622    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"]
Sep 10 21:38:37.239596 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:38:37.239572    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe792f0_3ba2_4a52_a92a_e8184c0c0e11.slice/crio-1b6e287f5a02bb162cb4fb42d56d322b5c7fc560ec824513515481a758687b44 WatchSource:0}: Error finding container 1b6e287f5a02bb162cb4fb42d56d322b5c7fc560ec824513515481a758687b44: Status 404 returned error can't find the container with id 1b6e287f5a02bb162cb4fb42d56d322b5c7fc560ec824513515481a758687b44
Sep 10 21:38:37.273984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:37.273964    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"]
Sep 10 21:38:37.276624 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:38:37.276601    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24aa610b_03fe_4eb4_a9f6_a80fbb89f8c9.slice/crio-1e811bdfb27198955c7ead94000232a7e7e6781daea7a547a0a5140ce0c44b83 WatchSource:0}: Error finding container 1e811bdfb27198955c7ead94000232a7e7e6781daea7a547a0a5140ce0c44b83: Status 404 returned error can't find the container with id 1e811bdfb27198955c7ead94000232a7e7e6781daea7a547a0a5140ce0c44b83
Sep 10 21:38:37.303640 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:37.303604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg" event={"ID":"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11","Type":"ContainerStarted","Data":"1b6e287f5a02bb162cb4fb42d56d322b5c7fc560ec824513515481a758687b44"}
Sep 10 21:38:37.305313 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:37.305289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8" event={"ID":"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9","Type":"ContainerStarted","Data":"1e811bdfb27198955c7ead94000232a7e7e6781daea7a547a0a5140ce0c44b83"}
Sep 10 21:38:38.312510 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:38.312461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg" event={"ID":"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11","Type":"ContainerStarted","Data":"65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c"}
Sep 10 21:38:38.312963 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:38.312533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:38:38.313989 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:38.313966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8" event={"ID":"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9","Type":"ContainerStarted","Data":"d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb"}
Sep 10 21:38:38.314114 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:38.314096    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:38:38.346009 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:38.345964    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8" podStartSLOduration=2.345949958 podStartE2EDuration="2.345949958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:38:38.345283493 +0000 UTC m=+1156.993197660" watchObservedRunningTime="2026-09-10 21:38:38.345949958 +0000 UTC m=+1156.993864145"
Sep 10 21:38:38.347389 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:38.347357    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg" podStartSLOduration=2.347342496 podStartE2EDuration="2.347342496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:38:38.328859295 +0000 UTC m=+1156.976773465" watchObservedRunningTime="2026-09-10 21:38:38.347342496 +0000 UTC m=+1156.995256719"
Sep 10 21:38:45.015520 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:45.015484    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:38:49.320889 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:49.320855    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:38:49.321312 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:38:49.321024    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:39:02.806790 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.806754    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"]
Sep 10 21:39:02.809307 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.809288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:02.812223 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.812201    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"default-dockercfg-6kjsn\""
Sep 10 21:39:02.812356 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.812206    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"auth-test-model-67d5e2-kserve-self-signed-certs\""
Sep 10 21:39:02.812356 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.812206    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"openshift-service-ca.crt\""
Sep 10 21:39:02.812356 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.812264    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"kube-root-ca.crt\""
Sep 10 21:39:02.821742 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.821712    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"]
Sep 10 21:39:02.959384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.959343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-dshm\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:02.959626 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.959393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53c66cc-8884-44bd-b841-e04f27f68e3e-tls-certs\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:02.959626 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.959506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-model-cache\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:02.959626 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.959606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-tmp-dir\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:02.959772 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.959642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7c48\" (UniqueName: \"kubernetes.io/projected/b53c66cc-8884-44bd-b841-e04f27f68e3e-kube-api-access-q7c48\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:02.959772 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:02.959736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-home\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.060283 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.060197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-home\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.060283 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.060240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-dshm\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.060283 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.060273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53c66cc-8884-44bd-b841-e04f27f68e3e-tls-certs\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.060613 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.060322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-model-cache\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.060613 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.060441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-tmp-dir\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.060613 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.060481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7c48\" (UniqueName: \"kubernetes.io/projected/b53c66cc-8884-44bd-b841-e04f27f68e3e-kube-api-access-q7c48\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.060762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.060636    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-home\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.060762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.060689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-model-cache\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.060872 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.060813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-tmp-dir\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.062746 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.062719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-dshm\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.062917 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.062900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53c66cc-8884-44bd-b841-e04f27f68e3e-tls-certs\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.069689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.069667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7c48\" (UniqueName: \"kubernetes.io/projected/b53c66cc-8884-44bd-b841-e04f27f68e3e-kube-api-access-q7c48\") pod \"auth-test-model-67d5e2-kserve-5795579b6c-4kdnh\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.120775 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.120748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:03.391668 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.391642    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"]
Sep 10 21:39:03.393848 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:39:03.393816    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb53c66cc_8884_44bd_b841_e04f27f68e3e.slice/crio-c1fd3c6ada1318b9fc4329b57d0e1ddfa115a27f26513db4719dc9019e785450 WatchSource:0}: Error finding container c1fd3c6ada1318b9fc4329b57d0e1ddfa115a27f26513db4719dc9019e785450: Status 404 returned error can't find the container with id c1fd3c6ada1318b9fc4329b57d0e1ddfa115a27f26513db4719dc9019e785450
Sep 10 21:39:03.445723 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:03.445696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh" event={"ID":"b53c66cc-8884-44bd-b841-e04f27f68e3e","Type":"ContainerStarted","Data":"c1fd3c6ada1318b9fc4329b57d0e1ddfa115a27f26513db4719dc9019e785450"}
Sep 10 21:39:04.452518 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:04.452477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh" event={"ID":"b53c66cc-8884-44bd-b841-e04f27f68e3e","Type":"ContainerStarted","Data":"a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1"}
Sep 10 21:39:04.468021 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:04.467973    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh" podStartSLOduration=2.467957944 podStartE2EDuration="2.467957944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:39:04.466872209 +0000 UTC m=+1183.114786405" watchObservedRunningTime="2026-09-10 21:39:04.467957944 +0000 UTC m=+1183.115872112"
Sep 10 21:39:07.314718 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:07.314679    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:39:13.121908 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:13.121870    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:13.121908 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:13.121919    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:13.134455 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:13.134405    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:13.512302 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:13.512266    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:39:25.614906 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:25.614858    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:39:49.106100 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:39:49.106063    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:40:03.807670 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:03.807630    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:40:15.286933 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.286842    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"]
Sep 10 21:40:15.287372 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.287117    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8" podUID="24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9" containerName="payload-processing" containerID="cri-o://d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb" gracePeriod=30
Sep 10 21:40:15.294921 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.294866    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"]
Sep 10 21:40:15.295785 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.295364    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg" podUID="3fe792f0-3ba2-4a52-a92a-e8184c0c0e11" containerName="payload-pre-processing" containerID="cri-o://65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c" gracePeriod=30
Sep 10 21:40:15.549764 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.549702    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:40:15.578737 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.578704    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:40:15.669014 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.668979    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcpll\" (UniqueName: \"kubernetes.io/projected/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-kube-api-access-kcpll\") pod \"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9\" (UID: \"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9\") "
Sep 10 21:40:15.669186 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.669041    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-plugins-config-volume\") pod \"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9\" (UID: \"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9\") "
Sep 10 21:40:15.669186 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.669065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdhjs\" (UniqueName: \"kubernetes.io/projected/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-kube-api-access-vdhjs\") pod \"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11\" (UID: \"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11\") "
Sep 10 21:40:15.669186 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.669180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-plugins-config-volume\") pod \"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11\" (UID: \"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11\") "
Sep 10 21:40:15.669538 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.669509    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9" (UID: "24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:40:15.669618 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.669553    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3fe792f0-3ba2-4a52-a92a-e8184c0c0e11" (UID: "3fe792f0-3ba2-4a52-a92a-e8184c0c0e11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:40:15.671366 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.671342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-kube-api-access-vdhjs" (OuterVolumeSpecName: "kube-api-access-vdhjs") pod "3fe792f0-3ba2-4a52-a92a-e8184c0c0e11" (UID: "3fe792f0-3ba2-4a52-a92a-e8184c0c0e11"). InnerVolumeSpecName "kube-api-access-vdhjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:40:15.671462 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.671362    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-kube-api-access-kcpll" (OuterVolumeSpecName: "kube-api-access-kcpll") pod "24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9" (UID: "24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9"). InnerVolumeSpecName "kube-api-access-kcpll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:40:15.770233 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.770199    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcpll\" (UniqueName: \"kubernetes.io/projected/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-kube-api-access-kcpll\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:40:15.770233 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.770228    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:40:15.770233 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.770239    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdhjs\" (UniqueName: \"kubernetes.io/projected/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-kube-api-access-vdhjs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:40:15.770524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.770248    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:40:15.825719 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.825629    2580 generic.go:356] "Generic (PLEG): container finished" podID="3fe792f0-3ba2-4a52-a92a-e8184c0c0e11" containerID="65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c" exitCode=0
Sep 10 21:40:15.825719 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.825671    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"
Sep 10 21:40:15.825965 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.825716    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg" event={"ID":"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11","Type":"ContainerDied","Data":"65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c"}
Sep 10 21:40:15.825965 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.825772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg" event={"ID":"3fe792f0-3ba2-4a52-a92a-e8184c0c0e11","Type":"ContainerDied","Data":"1b6e287f5a02bb162cb4fb42d56d322b5c7fc560ec824513515481a758687b44"}
Sep 10 21:40:15.825965 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.825800    2580 scope.go:117] "RemoveContainer" containerID="65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c"
Sep 10 21:40:15.828460 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.828397    2580 generic.go:356] "Generic (PLEG): container finished" podID="24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9" containerID="d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb" exitCode=0
Sep 10 21:40:15.828576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.828555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8" event={"ID":"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9","Type":"ContainerDied","Data":"d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb"}
Sep 10 21:40:15.828653 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.828634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8" event={"ID":"24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9","Type":"ContainerDied","Data":"1e811bdfb27198955c7ead94000232a7e7e6781daea7a547a0a5140ce0c44b83"}
Sep 10 21:40:15.828780 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.828764    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"
Sep 10 21:40:15.838529 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.838506    2580 scope.go:117] "RemoveContainer" containerID="65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c"
Sep 10 21:40:15.838835 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:40:15.838816    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c\": container with ID starting with 65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c not found: ID does not exist" containerID="65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c"
Sep 10 21:40:15.838930 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.838843    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c"} err="failed to get container status \"65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c\": rpc error: code = NotFound desc = could not find container \"65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c\": container with ID starting with 65e95311ec044bd1a7d6d202246475c00de12b33adb0b646d3388facf47a058c not found: ID does not exist"
Sep 10 21:40:15.838930 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.838859    2580 scope.go:117] "RemoveContainer" containerID="d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb"
Sep 10 21:40:15.847506 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.847336    2580 scope.go:117] "RemoveContainer" containerID="d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb"
Sep 10 21:40:15.847701 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:40:15.847673    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb\": container with ID starting with d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb not found: ID does not exist" containerID="d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb"
Sep 10 21:40:15.847753 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.847712    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb"} err="failed to get container status \"d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb\": rpc error: code = NotFound desc = could not find container \"d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb\": container with ID starting with d19fe53e4898dd729f60cec66099bfb2701cbad9ad37d0a5e0521ea2b69bd9eb not found: ID does not exist"
Sep 10 21:40:15.849486 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.849465    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"]
Sep 10 21:40:15.853534 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.853512    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8c245620-74c77fbf78-ghbvg"]
Sep 10 21:40:15.866378 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.866347    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"]
Sep 10 21:40:15.868274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:15.868241    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8c245620-5dc56c5fc-4bvt8"]
Sep 10 21:40:17.820745 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:17.820715    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9" path="/var/lib/kubelet/pods/24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9/volumes"
Sep 10 21:40:17.821140 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:17.821108    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe792f0-3ba2-4a52-a92a-e8184c0c0e11" path="/var/lib/kubelet/pods/3fe792f0-3ba2-4a52-a92a-e8184c0c0e11/volumes"
Sep 10 21:40:25.515334 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:25.515286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:40:36.599308 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:36.599263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:40:48.409916 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:40:48.409877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:41:00.416875 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:00.416830    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:41:16.003372 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:16.003336    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:41:21.410728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:21.410692    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"]
Sep 10 21:41:22.518468 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:22.518431    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"]
Sep 10 21:41:27.705239 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:27.705198    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:41:31.511312 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.511271    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"]
Sep 10 21:41:31.511896 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.511875    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9" containerName="payload-processing"
Sep 10 21:41:31.511979 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.511899    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9" containerName="payload-processing"
Sep 10 21:41:31.511979 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.511916    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe792f0-3ba2-4a52-a92a-e8184c0c0e11" containerName="payload-pre-processing"
Sep 10 21:41:31.511979 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.511923    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe792f0-3ba2-4a52-a92a-e8184c0c0e11" containerName="payload-pre-processing"
Sep 10 21:41:31.512140 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.512016    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe792f0-3ba2-4a52-a92a-e8184c0c0e11" containerName="payload-pre-processing"
Sep 10 21:41:31.512140 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.512038    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24aa610b-03fe-4eb4-a9f6-a80fbb89f8c9" containerName="payload-processing"
Sep 10 21:41:31.515232 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.515212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.517502 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.517477    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d9e68\"/\"sub-test-model-2d9e68-kserve-self-signed-certs\""
Sep 10 21:41:31.524649 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.524629    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"]
Sep 10 21:41:31.621763 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.621720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtdt\" (UniqueName: \"kubernetes.io/projected/7ff9edc6-532e-4286-bfd4-6980a1569c46-kube-api-access-zvtdt\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.621951 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.621797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-tmp-dir\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.621951 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.621825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-home\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.621951 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.621924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff9edc6-532e-4286-bfd4-6980a1569c46-tls-certs\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.622116 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.621969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-dshm\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.622116 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.621990    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-model-cache\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.722572 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.722537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff9edc6-532e-4286-bfd4-6980a1569c46-tls-certs\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.722762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.722582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-dshm\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.722762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.722600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-model-cache\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.722762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.722724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtdt\" (UniqueName: \"kubernetes.io/projected/7ff9edc6-532e-4286-bfd4-6980a1569c46-kube-api-access-zvtdt\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.722954 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.722789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-tmp-dir\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.723016 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.722976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-model-cache\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.723016 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.722977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-home\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.723155 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.723063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-tmp-dir\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.723231 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.723213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-home\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.725019 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.724996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-dshm\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.725288 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.725268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff9edc6-532e-4286-bfd4-6980a1569c46-tls-certs\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.730451 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.730405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtdt\" (UniqueName: \"kubernetes.io/projected/7ff9edc6-532e-4286-bfd4-6980a1569c46-kube-api-access-zvtdt\") pod \"sub-test-model-2d9e68-kserve-57d54895ff-25gcj\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:31.826242 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:31.826160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:32.099739 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:32.099714    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"]
Sep 10 21:41:32.101835 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:41:32.101808    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ff9edc6_532e_4286_bfd4_6980a1569c46.slice/crio-bc8d988cd15a58ed11090016460e564e160d01246d30018c4a4b63e3882aeb86 WatchSource:0}: Error finding container bc8d988cd15a58ed11090016460e564e160d01246d30018c4a4b63e3882aeb86: Status 404 returned error can't find the container with id bc8d988cd15a58ed11090016460e564e160d01246d30018c4a4b63e3882aeb86
Sep 10 21:41:32.216107 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:32.216075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj" event={"ID":"7ff9edc6-532e-4286-bfd4-6980a1569c46","Type":"ContainerStarted","Data":"1e36da55b51cc47419f5ddd3ee7f5548095d6b142f58818e679e35fa456721f5"}
Sep 10 21:41:32.216268 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:32.216119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj" event={"ID":"7ff9edc6-532e-4286-bfd4-6980a1569c46","Type":"ContainerStarted","Data":"bc8d988cd15a58ed11090016460e564e160d01246d30018c4a4b63e3882aeb86"}
Sep 10 21:41:32.234323 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:32.234274    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj" podStartSLOduration=1.23425762 podStartE2EDuration="1.23425762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:41:32.231727986 +0000 UTC m=+1330.879642175" watchObservedRunningTime="2026-09-10 21:41:32.23425762 +0000 UTC m=+1330.882171787"
Sep 10 21:41:36.429907 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.429856    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m" podUID="9453cde0-6173-43c3-861b-1738e2a6a4e0" containerName="main" containerID="cri-o://e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f" gracePeriod=45
Sep 10 21:41:36.684949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.684883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:41:36.776641 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.776601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-model-cache\") pod \"9453cde0-6173-43c3-861b-1738e2a6a4e0\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") "
Sep 10 21:41:36.776641 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.776644    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-dshm\") pod \"9453cde0-6173-43c3-861b-1738e2a6a4e0\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") "
Sep 10 21:41:36.776907 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.776684    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-home\") pod \"9453cde0-6173-43c3-861b-1738e2a6a4e0\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") "
Sep 10 21:41:36.776907 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.776719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bth76\" (UniqueName: \"kubernetes.io/projected/9453cde0-6173-43c3-861b-1738e2a6a4e0-kube-api-access-bth76\") pod \"9453cde0-6173-43c3-861b-1738e2a6a4e0\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") "
Sep 10 21:41:36.776907 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.776752    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9453cde0-6173-43c3-861b-1738e2a6a4e0-tls-certs\") pod \"9453cde0-6173-43c3-861b-1738e2a6a4e0\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") "
Sep 10 21:41:36.776907 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.776778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-tmp-dir\") pod \"9453cde0-6173-43c3-861b-1738e2a6a4e0\" (UID: \"9453cde0-6173-43c3-861b-1738e2a6a4e0\") "
Sep 10 21:41:36.777113 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.776976    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "9453cde0-6173-43c3-861b-1738e2a6a4e0" (UID: "9453cde0-6173-43c3-861b-1738e2a6a4e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:41:36.777113 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.776998    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-home" (OuterVolumeSpecName: "home") pod "9453cde0-6173-43c3-861b-1738e2a6a4e0" (UID: "9453cde0-6173-43c3-861b-1738e2a6a4e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:41:36.777279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.777101    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9453cde0-6173-43c3-861b-1738e2a6a4e0" (UID: "9453cde0-6173-43c3-861b-1738e2a6a4e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:41:36.777279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.777263    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:36.777368 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.777286    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:36.777368 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.777299    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:36.779074 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.779046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-dshm" (OuterVolumeSpecName: "dshm") pod "9453cde0-6173-43c3-861b-1738e2a6a4e0" (UID: "9453cde0-6173-43c3-861b-1738e2a6a4e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:41:36.779193 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.779093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9453cde0-6173-43c3-861b-1738e2a6a4e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9453cde0-6173-43c3-861b-1738e2a6a4e0" (UID: "9453cde0-6173-43c3-861b-1738e2a6a4e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:41:36.779193 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.779115    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9453cde0-6173-43c3-861b-1738e2a6a4e0-kube-api-access-bth76" (OuterVolumeSpecName: "kube-api-access-bth76") pod "9453cde0-6173-43c3-861b-1738e2a6a4e0" (UID: "9453cde0-6173-43c3-861b-1738e2a6a4e0"). InnerVolumeSpecName "kube-api-access-bth76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:41:36.878607 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.878566    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9453cde0-6173-43c3-861b-1738e2a6a4e0-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:36.878607 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.878599    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9453cde0-6173-43c3-861b-1738e2a6a4e0-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:36.878607 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:36.878611    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bth76\" (UniqueName: \"kubernetes.io/projected/9453cde0-6173-43c3-861b-1738e2a6a4e0-kube-api-access-bth76\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:37.243358 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.243318    2580 generic.go:356] "Generic (PLEG): container finished" podID="9453cde0-6173-43c3-861b-1738e2a6a4e0" containerID="e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f" exitCode=0
Sep 10 21:41:37.243591 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.243371    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"
Sep 10 21:41:37.243591 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.243393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m" event={"ID":"9453cde0-6173-43c3-861b-1738e2a6a4e0","Type":"ContainerDied","Data":"e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f"}
Sep 10 21:41:37.243591 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.243459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m" event={"ID":"9453cde0-6173-43c3-861b-1738e2a6a4e0","Type":"ContainerDied","Data":"e237b288a75882de2bf072ef98d405f5148c6fdf6e5475a8cd36ba86efd5a282"}
Sep 10 21:41:37.243591 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.243479    2580 scope.go:117] "RemoveContainer" containerID="e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f"
Sep 10 21:41:37.252805 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.252791    2580 scope.go:117] "RemoveContainer" containerID="e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f"
Sep 10 21:41:37.253071 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:41:37.253056    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f\": container with ID starting with e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f not found: ID does not exist" containerID="e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f"
Sep 10 21:41:37.253119 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.253080    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f"} err="failed to get container status \"e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f\": rpc error: code = NotFound desc = could not find container \"e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f\": container with ID starting with e85b31d8ecfbea6754305ebe61cee645aa1c710d92b749d6f54345a04857370f not found: ID does not exist"
Sep 10 21:41:37.263600 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.263393    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"]
Sep 10 21:41:37.265291 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.265272    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/auth-test-model-2d9e68-kserve-7866776f55-r6q4m"]
Sep 10 21:41:37.540244 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.540150    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh" podUID="b53c66cc-8884-44bd-b841-e04f27f68e3e" containerName="main" containerID="cri-o://a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1" gracePeriod=45
Sep 10 21:41:37.609951 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.609913    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:41:37.793192 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.793134    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:41:37.820955 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.820915    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9453cde0-6173-43c3-861b-1738e2a6a4e0" path="/var/lib/kubelet/pods/9453cde0-6173-43c3-861b-1738e2a6a4e0/volumes"
Sep 10 21:41:37.888353 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-model-cache\") pod \"b53c66cc-8884-44bd-b841-e04f27f68e3e\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") "
Sep 10 21:41:37.888576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888380    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-dshm\") pod \"b53c66cc-8884-44bd-b841-e04f27f68e3e\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") "
Sep 10 21:41:37.888576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888444    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-home\") pod \"b53c66cc-8884-44bd-b841-e04f27f68e3e\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") "
Sep 10 21:41:37.888576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53c66cc-8884-44bd-b841-e04f27f68e3e-tls-certs\") pod \"b53c66cc-8884-44bd-b841-e04f27f68e3e\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") "
Sep 10 21:41:37.888576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888541    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-tmp-dir\") pod \"b53c66cc-8884-44bd-b841-e04f27f68e3e\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") "
Sep 10 21:41:37.888576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888573    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7c48\" (UniqueName: \"kubernetes.io/projected/b53c66cc-8884-44bd-b841-e04f27f68e3e-kube-api-access-q7c48\") pod \"b53c66cc-8884-44bd-b841-e04f27f68e3e\" (UID: \"b53c66cc-8884-44bd-b841-e04f27f68e3e\") "
Sep 10 21:41:37.888859 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888691    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b53c66cc-8884-44bd-b841-e04f27f68e3e" (UID: "b53c66cc-8884-44bd-b841-e04f27f68e3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:41:37.888859 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888749    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-home" (OuterVolumeSpecName: "home") pod "b53c66cc-8884-44bd-b841-e04f27f68e3e" (UID: "b53c66cc-8884-44bd-b841-e04f27f68e3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:41:37.888859 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b53c66cc-8884-44bd-b841-e04f27f68e3e" (UID: "b53c66cc-8884-44bd-b841-e04f27f68e3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:41:37.888980 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888954    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:37.888980 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888972    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:37.889059 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.888984    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:37.890910 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.890881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-dshm" (OuterVolumeSpecName: "dshm") pod "b53c66cc-8884-44bd-b841-e04f27f68e3e" (UID: "b53c66cc-8884-44bd-b841-e04f27f68e3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:41:37.891030 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.890913    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b53c66cc-8884-44bd-b841-e04f27f68e3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b53c66cc-8884-44bd-b841-e04f27f68e3e" (UID: "b53c66cc-8884-44bd-b841-e04f27f68e3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:41:37.891030 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.890965    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53c66cc-8884-44bd-b841-e04f27f68e3e-kube-api-access-q7c48" (OuterVolumeSpecName: "kube-api-access-q7c48") pod "b53c66cc-8884-44bd-b841-e04f27f68e3e" (UID: "b53c66cc-8884-44bd-b841-e04f27f68e3e"). InnerVolumeSpecName "kube-api-access-q7c48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:41:37.990242 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.990109    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53c66cc-8884-44bd-b841-e04f27f68e3e-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:37.990242 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.990141    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53c66cc-8884-44bd-b841-e04f27f68e3e-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:37.990242 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:37.990156    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7c48\" (UniqueName: \"kubernetes.io/projected/b53c66cc-8884-44bd-b841-e04f27f68e3e-kube-api-access-q7c48\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:41:38.249824 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:38.249788    2580 generic.go:356] "Generic (PLEG): container finished" podID="b53c66cc-8884-44bd-b841-e04f27f68e3e" containerID="a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1" exitCode=0
Sep 10 21:41:38.250022 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:38.249846    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"
Sep 10 21:41:38.250022 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:38.249869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh" event={"ID":"b53c66cc-8884-44bd-b841-e04f27f68e3e","Type":"ContainerDied","Data":"a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1"}
Sep 10 21:41:38.250022 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:38.249906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh" event={"ID":"b53c66cc-8884-44bd-b841-e04f27f68e3e","Type":"ContainerDied","Data":"c1fd3c6ada1318b9fc4329b57d0e1ddfa115a27f26513db4719dc9019e785450"}
Sep 10 21:41:38.250022 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:38.249921    2580 scope.go:117] "RemoveContainer" containerID="a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1"
Sep 10 21:41:38.259047 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:38.259026    2580 scope.go:117] "RemoveContainer" containerID="a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1"
Sep 10 21:41:38.259294 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:41:38.259278    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1\": container with ID starting with a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1 not found: ID does not exist" containerID="a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1"
Sep 10 21:41:38.259347 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:38.259299    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1"} err="failed to get container status \"a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1\": rpc error: code = NotFound desc = could not find container \"a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1\": container with ID starting with a2d98ac28cef0ffd87a85fdb0283def706e8f51584501f4b198a6f8d1609f9f1 not found: ID does not exist"
Sep 10 21:41:38.271142 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:38.271116    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"]
Sep 10 21:41:38.279031 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:38.279007    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/auth-test-model-67d5e2-kserve-5795579b6c-4kdnh"]
Sep 10 21:41:39.820626 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:39.820592    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b53c66cc-8884-44bd-b841-e04f27f68e3e" path="/var/lib/kubelet/pods/b53c66cc-8884-44bd-b841-e04f27f68e3e/volumes"
Sep 10 21:41:41.826727 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:41.826693    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:41.826727 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:41.826738    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:41.839684 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:41.839658    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:42.283792 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:42.283760    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:41:54.014464 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.014430    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"]
Sep 10 21:41:54.014911 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.014843    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b53c66cc-8884-44bd-b841-e04f27f68e3e" containerName="main"
Sep 10 21:41:54.014911 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.014855    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b53c66cc-8884-44bd-b841-e04f27f68e3e" containerName="main"
Sep 10 21:41:54.014911 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.014870    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9453cde0-6173-43c3-861b-1738e2a6a4e0" containerName="main"
Sep 10 21:41:54.014911 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.014875    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9453cde0-6173-43c3-861b-1738e2a6a4e0" containerName="main"
Sep 10 21:41:54.015044 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.014949    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b53c66cc-8884-44bd-b841-e04f27f68e3e" containerName="main"
Sep 10 21:41:54.015044 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.014957    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9453cde0-6173-43c3-861b-1738e2a6a4e0" containerName="main"
Sep 10 21:41:54.018100 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.018079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.021263 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.021236    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"kube-root-ca.crt\""
Sep 10 21:41:54.021392 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.021297    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"openshift-service-ca.crt\""
Sep 10 21:41:54.021392 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.021321    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"sub-test-model-67d5e2-kserve-self-signed-certs\""
Sep 10 21:41:54.021392 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.021245    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"default-dockercfg-6kjsn\""
Sep 10 21:41:54.029471 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.029450    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"]
Sep 10 21:41:54.051454 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.051376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78c1e697-6baf-4491-bb81-48b1a96d3314-tls-certs\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.051454 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.051449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrz7\" (UniqueName: \"kubernetes.io/projected/78c1e697-6baf-4491-bb81-48b1a96d3314-kube-api-access-gbrz7\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.051652 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.051510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-home\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.051652 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.051567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-tmp-dir\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.051652 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.051611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-model-cache\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.051652 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.051638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-dshm\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.152677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.152649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-home\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.152854 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.152707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-tmp-dir\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.152854 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.152747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-model-cache\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.152854 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.152772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-dshm\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.152854 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.152821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78c1e697-6baf-4491-bb81-48b1a96d3314-tls-certs\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.152854 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.152842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrz7\" (UniqueName: \"kubernetes.io/projected/78c1e697-6baf-4491-bb81-48b1a96d3314-kube-api-access-gbrz7\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.153143 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.153117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-home\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.153207 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.153139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-tmp-dir\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.153207 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.153176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-model-cache\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.155309 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.155284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-dshm\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.155560 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.155543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78c1e697-6baf-4491-bb81-48b1a96d3314-tls-certs\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.161472 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.161454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrz7\" (UniqueName: \"kubernetes.io/projected/78c1e697-6baf-4491-bb81-48b1a96d3314-kube-api-access-gbrz7\") pod \"sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.328526 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.328440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:41:54.601269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:54.601245    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"]
Sep 10 21:41:54.603063 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:41:54.603037    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78c1e697_6baf_4491_bb81_48b1a96d3314.slice/crio-ea56be24aa077fce41376b246a3b8b887d3999d280d06b970967d7e49c3f582c WatchSource:0}: Error finding container ea56be24aa077fce41376b246a3b8b887d3999d280d06b970967d7e49c3f582c: Status 404 returned error can't find the container with id ea56be24aa077fce41376b246a3b8b887d3999d280d06b970967d7e49c3f582c
Sep 10 21:41:55.350348 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:55.350314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw" event={"ID":"78c1e697-6baf-4491-bb81-48b1a96d3314","Type":"ContainerStarted","Data":"37fb9f044c4ab7675c9362cf3badc40abb54cc2db0d56eb049a93d35a0b2aeb6"}
Sep 10 21:41:55.350348 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:55.350356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw" event={"ID":"78c1e697-6baf-4491-bb81-48b1a96d3314","Type":"ContainerStarted","Data":"ea56be24aa077fce41376b246a3b8b887d3999d280d06b970967d7e49c3f582c"}
Sep 10 21:41:55.367811 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:55.367757    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw" podStartSLOduration=2.367733746 podStartE2EDuration="2.367733746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:41:55.366221813 +0000 UTC m=+1354.014136007" watchObservedRunningTime="2026-09-10 21:41:55.367733746 +0000 UTC m=+1354.015647915"
Sep 10 21:41:58.912828 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:41:58.912790    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:42:04.328773 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:04.328736    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:42:04.329200 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:04.328791    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:42:04.341318 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:04.341287    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:42:04.407339 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:04.407308    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:42:06.571453 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.571389    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"]
Sep 10 21:42:06.576356 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.576332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:42:06.578387 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.578364    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-93f046\""
Sep 10 21:42:06.578539 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.578522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-93f046-dockercfg-2ffj2\""
Sep 10 21:42:06.585950 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.585922    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"]
Sep 10 21:42:06.619596 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.619560    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"]
Sep 10 21:42:06.623166 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.623143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:42:06.632712 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.632682    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"]
Sep 10 21:42:06.683288 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.683248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vldvr\" (UniqueName: \"kubernetes.io/projected/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-kube-api-access-vldvr\") pod \"payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5\" (UID: \"f2594c81-0d32-448a-bcbe-2d1b7d1b3690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:42:06.683489 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.683316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5\" (UID: \"f2594c81-0d32-448a-bcbe-2d1b7d1b3690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:42:06.784511 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.784465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5\" (UID: \"f2594c81-0d32-448a-bcbe-2d1b7d1b3690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:42:06.784511 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.784519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cxhh\" (UniqueName: \"kubernetes.io/projected/8693d567-a118-4ce3-a2f8-4c13d7f9b197-kube-api-access-5cxhh\") pod \"payload-processing-e2e-api-a-93f046-576697f787-z9mks\" (UID: \"8693d567-a118-4ce3-a2f8-4c13d7f9b197\") " pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:42:06.784734 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.784629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8693d567-a118-4ce3-a2f8-4c13d7f9b197-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93f046-576697f787-z9mks\" (UID: \"8693d567-a118-4ce3-a2f8-4c13d7f9b197\") " pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:42:06.784780 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.784758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vldvr\" (UniqueName: \"kubernetes.io/projected/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-kube-api-access-vldvr\") pod \"payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5\" (UID: \"f2594c81-0d32-448a-bcbe-2d1b7d1b3690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:42:06.785273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.785255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5\" (UID: \"f2594c81-0d32-448a-bcbe-2d1b7d1b3690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:42:06.792319 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.792288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vldvr\" (UniqueName: \"kubernetes.io/projected/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-kube-api-access-vldvr\") pod \"payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5\" (UID: \"f2594c81-0d32-448a-bcbe-2d1b7d1b3690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:42:06.886255 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.886155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxhh\" (UniqueName: \"kubernetes.io/projected/8693d567-a118-4ce3-a2f8-4c13d7f9b197-kube-api-access-5cxhh\") pod \"payload-processing-e2e-api-a-93f046-576697f787-z9mks\" (UID: \"8693d567-a118-4ce3-a2f8-4c13d7f9b197\") " pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:42:06.886255 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.886201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8693d567-a118-4ce3-a2f8-4c13d7f9b197-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93f046-576697f787-z9mks\" (UID: \"8693d567-a118-4ce3-a2f8-4c13d7f9b197\") " pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:42:06.886944 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.886924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8693d567-a118-4ce3-a2f8-4c13d7f9b197-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93f046-576697f787-z9mks\" (UID: \"8693d567-a118-4ce3-a2f8-4c13d7f9b197\") " pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:42:06.887949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.887929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:42:06.894587 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.894559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cxhh\" (UniqueName: \"kubernetes.io/projected/8693d567-a118-4ce3-a2f8-4c13d7f9b197-kube-api-access-5cxhh\") pod \"payload-processing-e2e-api-a-93f046-576697f787-z9mks\" (UID: \"8693d567-a118-4ce3-a2f8-4c13d7f9b197\") " pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:42:06.934568 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:06.934527    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:42:07.162007 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:07.161984    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"]
Sep 10 21:42:07.164117 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:42:07.164087    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2594c81_0d32_448a_bcbe_2d1b7d1b3690.slice/crio-17a9c13221e5ac906366dff1e6ac9f860f3612c38d4ee6486d7e6407c2c7afb6 WatchSource:0}: Error finding container 17a9c13221e5ac906366dff1e6ac9f860f3612c38d4ee6486d7e6407c2c7afb6: Status 404 returned error can't find the container with id 17a9c13221e5ac906366dff1e6ac9f860f3612c38d4ee6486d7e6407c2c7afb6
Sep 10 21:42:07.233911 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:07.233885    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"]
Sep 10 21:42:07.238130 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:42:07.238097    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8693d567_a118_4ce3_a2f8_4c13d7f9b197.slice/crio-80f537cb9fcad52aed0e9d768a6083b0665b622152ac5df17b27ccaf11f61d88 WatchSource:0}: Error finding container 80f537cb9fcad52aed0e9d768a6083b0665b622152ac5df17b27ccaf11f61d88: Status 404 returned error can't find the container with id 80f537cb9fcad52aed0e9d768a6083b0665b622152ac5df17b27ccaf11f61d88
Sep 10 21:42:07.423189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:07.423081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5" event={"ID":"f2594c81-0d32-448a-bcbe-2d1b7d1b3690","Type":"ContainerStarted","Data":"c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e"}
Sep 10 21:42:07.423189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:07.423149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5" event={"ID":"f2594c81-0d32-448a-bcbe-2d1b7d1b3690","Type":"ContainerStarted","Data":"17a9c13221e5ac906366dff1e6ac9f860f3612c38d4ee6486d7e6407c2c7afb6"}
Sep 10 21:42:07.423189 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:07.423179    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:42:07.424841 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:07.424817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks" event={"ID":"8693d567-a118-4ce3-a2f8-4c13d7f9b197","Type":"ContainerStarted","Data":"c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad"}
Sep 10 21:42:07.425052 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:07.424855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks" event={"ID":"8693d567-a118-4ce3-a2f8-4c13d7f9b197","Type":"ContainerStarted","Data":"80f537cb9fcad52aed0e9d768a6083b0665b622152ac5df17b27ccaf11f61d88"}
Sep 10 21:42:07.425052 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:07.424938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:42:07.439659 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:07.439609    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5" podStartSLOduration=1.439593747 podStartE2EDuration="1.439593747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:42:07.437379532 +0000 UTC m=+1366.085293701" watchObservedRunningTime="2026-09-10 21:42:07.439593747 +0000 UTC m=+1366.087507914"
Sep 10 21:42:07.452654 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:07.452601    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks" podStartSLOduration=1.452585029 podStartE2EDuration="1.452585029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:42:07.451756243 +0000 UTC m=+1366.099670408" watchObservedRunningTime="2026-09-10 21:42:07.452585029 +0000 UTC m=+1366.100499225"
Sep 10 21:42:15.911807 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:15.911772    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:42:18.431643 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:18.431609    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:42:18.432213 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:18.432109    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:42:35.116565 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:35.116525    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:42:35.714505 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:35.714471    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"]
Sep 10 21:42:35.719222 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:35.719202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:42:35.721352 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:35.721326    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a867da\""
Sep 10 21:42:35.730383 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:35.730357    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"]
Sep 10 21:42:35.756677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:35.756650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9\" (UID: \"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:42:35.756800 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:35.756711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwg2n\" (UniqueName: \"kubernetes.io/projected/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-kube-api-access-dwg2n\") pod \"payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9\" (UID: \"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:42:35.857345 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:35.857302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9\" (UID: \"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:42:35.857573 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:35.857365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwg2n\" (UniqueName: \"kubernetes.io/projected/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-kube-api-access-dwg2n\") pod \"payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9\" (UID: \"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:42:35.858344 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:35.858313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9\" (UID: \"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:42:35.866385 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:35.866357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwg2n\" (UniqueName: \"kubernetes.io/projected/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-kube-api-access-dwg2n\") pod \"payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9\" (UID: \"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:42:36.030619 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:36.030517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:42:36.318823 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:36.318797    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"]
Sep 10 21:42:36.320941 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:42:36.320914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2187fe6_00cd_4cc4_ad3a_4d1c2219fcda.slice/crio-bc203a903e84fbb3ce077bbbee434bfbd6208ed7754d8851457180bb8959ef0c WatchSource:0}: Error finding container bc203a903e84fbb3ce077bbbee434bfbd6208ed7754d8851457180bb8959ef0c: Status 404 returned error can't find the container with id bc203a903e84fbb3ce077bbbee434bfbd6208ed7754d8851457180bb8959ef0c
Sep 10 21:42:36.583617 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:36.583514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9" event={"ID":"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda","Type":"ContainerStarted","Data":"ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b"}
Sep 10 21:42:36.583617 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:36.583581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9" event={"ID":"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda","Type":"ContainerStarted","Data":"bc203a903e84fbb3ce077bbbee434bfbd6208ed7754d8851457180bb8959ef0c"}
Sep 10 21:42:36.583842 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:36.583720    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:42:36.599890 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:36.599837    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9" podStartSLOduration=1.5998202849999998 podStartE2EDuration="1.599820285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:42:36.59862963 +0000 UTC m=+1395.246543822" watchObservedRunningTime="2026-09-10 21:42:36.599820285 +0000 UTC m=+1395.247734453"
Sep 10 21:42:47.592218 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:47.592178    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:42:54.604238 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:54.604203    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"]
Sep 10 21:42:55.512990 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:55.512954    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"]
Sep 10 21:42:57.512874 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:57.512829    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:42:58.401567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.401532    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"]
Sep 10 21:42:58.405206 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.405185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.407456 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.407433    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2d9e68\"/\"rate-test-model-2d9e68-kserve-self-signed-certs\""
Sep 10 21:42:58.413728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.413705    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"]
Sep 10 21:42:58.586819 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.586781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tmp-dir\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.587252 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.586832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-model-cache\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.587252 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.586880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-dshm\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.587252 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.586909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tls-certs\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.587252 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.586960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptmjx\" (UniqueName: \"kubernetes.io/projected/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-kube-api-access-ptmjx\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.587252 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.586987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-home\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.687910 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.687825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptmjx\" (UniqueName: \"kubernetes.io/projected/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-kube-api-access-ptmjx\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.687910 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.687862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-home\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.688132 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.687993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tmp-dir\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.688132 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.688034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-model-cache\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.688132 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.688068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-dshm\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.688132 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.688091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tls-certs\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.688360 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.688344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-home\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.688449 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.688397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tmp-dir\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.688449 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.688406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-model-cache\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.690636 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.690609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-dshm\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.690727 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.690644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tls-certs\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.695334 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.695314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptmjx\" (UniqueName: \"kubernetes.io/projected/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-kube-api-access-ptmjx\") pod \"rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") " pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.716790 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.716763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:42:58.985455 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:58.985432    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"]
Sep 10 21:42:58.988137 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:42:58.988110    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6a6a4bc_be9c_457d_95c1_a7d89d8eb173.slice/crio-8f0287c04d3022b77b3e42c0c061a6d9a74a28c0c3aff7f74a749ae478d78cc4 WatchSource:0}: Error finding container 8f0287c04d3022b77b3e42c0c061a6d9a74a28c0c3aff7f74a749ae478d78cc4: Status 404 returned error can't find the container with id 8f0287c04d3022b77b3e42c0c061a6d9a74a28c0c3aff7f74a749ae478d78cc4
Sep 10 21:42:59.705233 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:59.705200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n" event={"ID":"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173","Type":"ContainerStarted","Data":"20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511"}
Sep 10 21:42:59.705594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:59.705245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n" event={"ID":"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173","Type":"ContainerStarted","Data":"8f0287c04d3022b77b3e42c0c061a6d9a74a28c0c3aff7f74a749ae478d78cc4"}
Sep 10 21:42:59.722746 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:42:59.722700    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n" podStartSLOduration=1.7226851600000002 podStartE2EDuration="1.72268516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:42:59.721665931 +0000 UTC m=+1418.369580097" watchObservedRunningTime="2026-09-10 21:42:59.72268516 +0000 UTC m=+1418.370599332"
Sep 10 21:43:08.717852 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:08.717819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:43:08.718442 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:08.717866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:43:08.730032 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:08.730013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:43:08.763757 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:08.763728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:43:09.625783 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.625726    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj" podUID="7ff9edc6-532e-4286-bfd4-6980a1569c46" containerName="main" containerID="cri-o://1e36da55b51cc47419f5ddd3ee7f5548095d6b142f58818e679e35fa456721f5" gracePeriod=45
Sep 10 21:43:09.761109 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.761080    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ff9edc6-532e-4286-bfd4-6980a1569c46" containerID="1e36da55b51cc47419f5ddd3ee7f5548095d6b142f58818e679e35fa456721f5" exitCode=0
Sep 10 21:43:09.761537 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.761158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj" event={"ID":"7ff9edc6-532e-4286-bfd4-6980a1569c46","Type":"ContainerDied","Data":"1e36da55b51cc47419f5ddd3ee7f5548095d6b142f58818e679e35fa456721f5"}
Sep 10 21:43:09.875952 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.875896    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:43:09.890787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.890764    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-dshm\") pod \"7ff9edc6-532e-4286-bfd4-6980a1569c46\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") "
Sep 10 21:43:09.890941 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.890817    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff9edc6-532e-4286-bfd4-6980a1569c46-tls-certs\") pod \"7ff9edc6-532e-4286-bfd4-6980a1569c46\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") "
Sep 10 21:43:09.890941 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.890896    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-model-cache\") pod \"7ff9edc6-532e-4286-bfd4-6980a1569c46\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") "
Sep 10 21:43:09.891110 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.891087    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-home\") pod \"7ff9edc6-532e-4286-bfd4-6980a1569c46\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") "
Sep 10 21:43:09.891185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.891130    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvtdt\" (UniqueName: \"kubernetes.io/projected/7ff9edc6-532e-4286-bfd4-6980a1569c46-kube-api-access-zvtdt\") pod \"7ff9edc6-532e-4286-bfd4-6980a1569c46\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") "
Sep 10 21:43:09.891185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.891154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-tmp-dir\") pod \"7ff9edc6-532e-4286-bfd4-6980a1569c46\" (UID: \"7ff9edc6-532e-4286-bfd4-6980a1569c46\") "
Sep 10 21:43:09.891698 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.891352    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ff9edc6-532e-4286-bfd4-6980a1569c46" (UID: "7ff9edc6-532e-4286-bfd4-6980a1569c46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:43:09.891698 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.891498    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-home" (OuterVolumeSpecName: "home") pod "7ff9edc6-532e-4286-bfd4-6980a1569c46" (UID: "7ff9edc6-532e-4286-bfd4-6980a1569c46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:43:09.892611 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.892065    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ff9edc6-532e-4286-bfd4-6980a1569c46" (UID: "7ff9edc6-532e-4286-bfd4-6980a1569c46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:43:09.892611 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.892439    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:09.892611 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.892463    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:09.892611 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.892478    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:09.894254 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.894015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff9edc6-532e-4286-bfd4-6980a1569c46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ff9edc6-532e-4286-bfd4-6980a1569c46" (UID: "7ff9edc6-532e-4286-bfd4-6980a1569c46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:43:09.894254 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.894070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-dshm" (OuterVolumeSpecName: "dshm") pod "7ff9edc6-532e-4286-bfd4-6980a1569c46" (UID: "7ff9edc6-532e-4286-bfd4-6980a1569c46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:43:09.894651 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.894630    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff9edc6-532e-4286-bfd4-6980a1569c46-kube-api-access-zvtdt" (OuterVolumeSpecName: "kube-api-access-zvtdt") pod "7ff9edc6-532e-4286-bfd4-6980a1569c46" (UID: "7ff9edc6-532e-4286-bfd4-6980a1569c46"). InnerVolumeSpecName "kube-api-access-zvtdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:43:09.993545 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.993504    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff9edc6-532e-4286-bfd4-6980a1569c46-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:09.993545 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.993538    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff9edc6-532e-4286-bfd4-6980a1569c46-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:09.993545 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:09.993551    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvtdt\" (UniqueName: \"kubernetes.io/projected/7ff9edc6-532e-4286-bfd4-6980a1569c46-kube-api-access-zvtdt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:10.533877 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.533818    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw" podUID="78c1e697-6baf-4491-bb81-48b1a96d3314" containerName="main" containerID="cri-o://37fb9f044c4ab7675c9362cf3badc40abb54cc2db0d56eb049a93d35a0b2aeb6" gracePeriod=45
Sep 10 21:43:10.769313 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.769286    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"
Sep 10 21:43:10.769831 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.769308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj" event={"ID":"7ff9edc6-532e-4286-bfd4-6980a1569c46","Type":"ContainerDied","Data":"bc8d988cd15a58ed11090016460e564e160d01246d30018c4a4b63e3882aeb86"}
Sep 10 21:43:10.769831 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.769367    2580 scope.go:117] "RemoveContainer" containerID="1e36da55b51cc47419f5ddd3ee7f5548095d6b142f58818e679e35fa456721f5"
Sep 10 21:43:10.771707 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.771679    2580 generic.go:356] "Generic (PLEG): container finished" podID="78c1e697-6baf-4491-bb81-48b1a96d3314" containerID="37fb9f044c4ab7675c9362cf3badc40abb54cc2db0d56eb049a93d35a0b2aeb6" exitCode=0
Sep 10 21:43:10.771829 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.771760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw" event={"ID":"78c1e697-6baf-4491-bb81-48b1a96d3314","Type":"ContainerDied","Data":"37fb9f044c4ab7675c9362cf3badc40abb54cc2db0d56eb049a93d35a0b2aeb6"}
Sep 10 21:43:10.797859 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.797841    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:43:10.807259 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.807235    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"]
Sep 10 21:43:10.811041 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.811016    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/sub-test-model-2d9e68-kserve-57d54895ff-25gcj"]
Sep 10 21:43:10.902697 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.902654    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-model-cache\") pod \"78c1e697-6baf-4491-bb81-48b1a96d3314\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") "
Sep 10 21:43:10.902697 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.902704    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbrz7\" (UniqueName: \"kubernetes.io/projected/78c1e697-6baf-4491-bb81-48b1a96d3314-kube-api-access-gbrz7\") pod \"78c1e697-6baf-4491-bb81-48b1a96d3314\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") "
Sep 10 21:43:10.902960 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.902753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-home\") pod \"78c1e697-6baf-4491-bb81-48b1a96d3314\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") "
Sep 10 21:43:10.902960 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.902813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-dshm\") pod \"78c1e697-6baf-4491-bb81-48b1a96d3314\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") "
Sep 10 21:43:10.902960 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.902876    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78c1e697-6baf-4491-bb81-48b1a96d3314-tls-certs\") pod \"78c1e697-6baf-4491-bb81-48b1a96d3314\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") "
Sep 10 21:43:10.902960 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.902910    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-tmp-dir\") pod \"78c1e697-6baf-4491-bb81-48b1a96d3314\" (UID: \"78c1e697-6baf-4491-bb81-48b1a96d3314\") "
Sep 10 21:43:10.903154 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.902994    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-model-cache" (OuterVolumeSpecName: "model-cache") pod "78c1e697-6baf-4491-bb81-48b1a96d3314" (UID: "78c1e697-6baf-4491-bb81-48b1a96d3314"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:43:10.903154 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.903053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-home" (OuterVolumeSpecName: "home") pod "78c1e697-6baf-4491-bb81-48b1a96d3314" (UID: "78c1e697-6baf-4491-bb81-48b1a96d3314"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:43:10.903279 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.903251    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78c1e697-6baf-4491-bb81-48b1a96d3314" (UID: "78c1e697-6baf-4491-bb81-48b1a96d3314"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:43:10.903334 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.903318    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:10.903393 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.903333    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:10.905109 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.905081    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c1e697-6baf-4491-bb81-48b1a96d3314-kube-api-access-gbrz7" (OuterVolumeSpecName: "kube-api-access-gbrz7") pod "78c1e697-6baf-4491-bb81-48b1a96d3314" (UID: "78c1e697-6baf-4491-bb81-48b1a96d3314"). InnerVolumeSpecName "kube-api-access-gbrz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:43:10.905228 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.905104    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-dshm" (OuterVolumeSpecName: "dshm") pod "78c1e697-6baf-4491-bb81-48b1a96d3314" (UID: "78c1e697-6baf-4491-bb81-48b1a96d3314"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:43:10.905228 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:10.905164    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c1e697-6baf-4491-bb81-48b1a96d3314-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78c1e697-6baf-4491-bb81-48b1a96d3314" (UID: "78c1e697-6baf-4491-bb81-48b1a96d3314"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:43:11.004173 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:11.004132    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:11.004173 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:11.004165    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbrz7\" (UniqueName: \"kubernetes.io/projected/78c1e697-6baf-4491-bb81-48b1a96d3314-kube-api-access-gbrz7\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:11.004173 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:11.004175    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78c1e697-6baf-4491-bb81-48b1a96d3314-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:11.004173 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:11.004185    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78c1e697-6baf-4491-bb81-48b1a96d3314-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:11.779672 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:11.779634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw" event={"ID":"78c1e697-6baf-4491-bb81-48b1a96d3314","Type":"ContainerDied","Data":"ea56be24aa077fce41376b246a3b8b887d3999d280d06b970967d7e49c3f582c"}
Sep 10 21:43:11.779672 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:11.779673    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"
Sep 10 21:43:11.780139 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:11.779689    2580 scope.go:117] "RemoveContainer" containerID="37fb9f044c4ab7675c9362cf3badc40abb54cc2db0d56eb049a93d35a0b2aeb6"
Sep 10 21:43:11.803326 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:11.803190    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"]
Sep 10 21:43:11.805277 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:11.805255    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/sub-test-model-67d5e2-kserve-5cdc4c9fb5-xdwjw"]
Sep 10 21:43:11.821991 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:11.821963    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c1e697-6baf-4491-bb81-48b1a96d3314" path="/var/lib/kubelet/pods/78c1e697-6baf-4491-bb81-48b1a96d3314/volumes"
Sep 10 21:43:11.822355 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:11.822342    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ff9edc6-532e-4286-bfd4-6980a1569c46" path="/var/lib/kubelet/pods/7ff9edc6-532e-4286-bfd4-6980a1569c46/volumes"
Sep 10 21:43:14.508751 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:14.508721    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:43:17.465592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.465550    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"]
Sep 10 21:43:17.466690 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.465835    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks" podUID="8693d567-a118-4ce3-a2f8-4c13d7f9b197" containerName="payload-processing" containerID="cri-o://c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad" gracePeriod=30
Sep 10 21:43:17.469026 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.468996    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"]
Sep 10 21:43:17.469286 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.469239    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5" podUID="f2594c81-0d32-448a-bcbe-2d1b7d1b3690" containerName="payload-pre-processing" containerID="cri-o://c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e" gracePeriod=30
Sep 10 21:43:17.759210 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.759174    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:43:17.761929 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.761908    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:43:17.814761 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.814717    2580 generic.go:356] "Generic (PLEG): container finished" podID="f2594c81-0d32-448a-bcbe-2d1b7d1b3690" containerID="c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e" exitCode=0
Sep 10 21:43:17.814951 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.814809    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"
Sep 10 21:43:17.816772 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.816740    2580 generic.go:356] "Generic (PLEG): container finished" podID="8693d567-a118-4ce3-a2f8-4c13d7f9b197" containerID="c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad" exitCode=0
Sep 10 21:43:17.816918 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.816821    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"
Sep 10 21:43:17.827505 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.827477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5" event={"ID":"f2594c81-0d32-448a-bcbe-2d1b7d1b3690","Type":"ContainerDied","Data":"c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e"}
Sep 10 21:43:17.827670 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.827539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5" event={"ID":"f2594c81-0d32-448a-bcbe-2d1b7d1b3690","Type":"ContainerDied","Data":"17a9c13221e5ac906366dff1e6ac9f860f3612c38d4ee6486d7e6407c2c7afb6"}
Sep 10 21:43:17.827670 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.827560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks" event={"ID":"8693d567-a118-4ce3-a2f8-4c13d7f9b197","Type":"ContainerDied","Data":"c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad"}
Sep 10 21:43:17.827670 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.827577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks" event={"ID":"8693d567-a118-4ce3-a2f8-4c13d7f9b197","Type":"ContainerDied","Data":"80f537cb9fcad52aed0e9d768a6083b0665b622152ac5df17b27ccaf11f61d88"}
Sep 10 21:43:17.827670 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.827604    2580 scope.go:117] "RemoveContainer" containerID="c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e"
Sep 10 21:43:17.837238 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.837216    2580 scope.go:117] "RemoveContainer" containerID="c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e"
Sep 10 21:43:17.837547 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:43:17.837529    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e\": container with ID starting with c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e not found: ID does not exist" containerID="c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e"
Sep 10 21:43:17.837623 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.837554    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e"} err="failed to get container status \"c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e\": rpc error: code = NotFound desc = could not find container \"c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e\": container with ID starting with c28649fcb65a2be09865ce29af67cfe7f089d1a940afe10576726c4b0d70cb4e not found: ID does not exist"
Sep 10 21:43:17.837623 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.837569    2580 scope.go:117] "RemoveContainer" containerID="c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad"
Sep 10 21:43:17.845463 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.845447    2580 scope.go:117] "RemoveContainer" containerID="c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad"
Sep 10 21:43:17.845738 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:43:17.845719    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad\": container with ID starting with c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad not found: ID does not exist" containerID="c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad"
Sep 10 21:43:17.845804 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.845760    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad"} err="failed to get container status \"c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad\": rpc error: code = NotFound desc = could not find container \"c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad\": container with ID starting with c7450073a448323a7870e054d46199cf4fe63f7c7dc4831a393ed4d41ad389ad not found: ID does not exist"
Sep 10 21:43:17.869139 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.869105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-plugins-config-volume\") pod \"f2594c81-0d32-448a-bcbe-2d1b7d1b3690\" (UID: \"f2594c81-0d32-448a-bcbe-2d1b7d1b3690\") "
Sep 10 21:43:17.869265 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.869169    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cxhh\" (UniqueName: \"kubernetes.io/projected/8693d567-a118-4ce3-a2f8-4c13d7f9b197-kube-api-access-5cxhh\") pod \"8693d567-a118-4ce3-a2f8-4c13d7f9b197\" (UID: \"8693d567-a118-4ce3-a2f8-4c13d7f9b197\") "
Sep 10 21:43:17.869265 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.869228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8693d567-a118-4ce3-a2f8-4c13d7f9b197-plugins-config-volume\") pod \"8693d567-a118-4ce3-a2f8-4c13d7f9b197\" (UID: \"8693d567-a118-4ce3-a2f8-4c13d7f9b197\") "
Sep 10 21:43:17.869334 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.869273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vldvr\" (UniqueName: \"kubernetes.io/projected/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-kube-api-access-vldvr\") pod \"f2594c81-0d32-448a-bcbe-2d1b7d1b3690\" (UID: \"f2594c81-0d32-448a-bcbe-2d1b7d1b3690\") "
Sep 10 21:43:17.869685 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.869650    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2594c81-0d32-448a-bcbe-2d1b7d1b3690" (UID: "f2594c81-0d32-448a-bcbe-2d1b7d1b3690"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:43:17.869685 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.869666    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8693d567-a118-4ce3-a2f8-4c13d7f9b197-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8693d567-a118-4ce3-a2f8-4c13d7f9b197" (UID: "8693d567-a118-4ce3-a2f8-4c13d7f9b197"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:43:17.871639 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.871617    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-kube-api-access-vldvr" (OuterVolumeSpecName: "kube-api-access-vldvr") pod "f2594c81-0d32-448a-bcbe-2d1b7d1b3690" (UID: "f2594c81-0d32-448a-bcbe-2d1b7d1b3690"). InnerVolumeSpecName "kube-api-access-vldvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:43:17.871723 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.871689    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8693d567-a118-4ce3-a2f8-4c13d7f9b197-kube-api-access-5cxhh" (OuterVolumeSpecName: "kube-api-access-5cxhh") pod "8693d567-a118-4ce3-a2f8-4c13d7f9b197" (UID: "8693d567-a118-4ce3-a2f8-4c13d7f9b197"). InnerVolumeSpecName "kube-api-access-5cxhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:43:17.970215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.970178    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:17.970215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.970208    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cxhh\" (UniqueName: \"kubernetes.io/projected/8693d567-a118-4ce3-a2f8-4c13d7f9b197-kube-api-access-5cxhh\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:17.970215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.970219    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8693d567-a118-4ce3-a2f8-4c13d7f9b197-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:17.970510 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:17.970228    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vldvr\" (UniqueName: \"kubernetes.io/projected/f2594c81-0d32-448a-bcbe-2d1b7d1b3690-kube-api-access-vldvr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:18.139976 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:18.139932    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"]
Sep 10 21:43:18.143847 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:18.143823    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93f046-7f9c4d8dd6-j96p5"]
Sep 10 21:43:18.155383 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:18.155361    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"]
Sep 10 21:43:18.161701 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:18.161680    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93f046-576697f787-z9mks"]
Sep 10 21:43:19.821434 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:19.821384    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8693d567-a118-4ce3-a2f8-4c13d7f9b197" path="/var/lib/kubelet/pods/8693d567-a118-4ce3-a2f8-4c13d7f9b197/volumes"
Sep 10 21:43:19.821809 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:19.821782    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2594c81-0d32-448a-bcbe-2d1b7d1b3690" path="/var/lib/kubelet/pods/f2594c81-0d32-448a-bcbe-2d1b7d1b3690/volumes"
Sep 10 21:43:21.948428 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:21.948390    2580 scope.go:117] "RemoveContainer" containerID="dcdecf942bba27c3df0d9925a5d4ac0043d06e56a2013a83ecf35512edb8933a"
Sep 10 21:43:30.510762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:30.510728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:43:38.104657 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.104621    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"]
Sep 10 21:43:38.105200 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105179    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c1e697-6baf-4491-bb81-48b1a96d3314" containerName="main"
Sep 10 21:43:38.105298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105203    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c1e697-6baf-4491-bb81-48b1a96d3314" containerName="main"
Sep 10 21:43:38.105298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105228    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff9edc6-532e-4286-bfd4-6980a1569c46" containerName="main"
Sep 10 21:43:38.105298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105237    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff9edc6-532e-4286-bfd4-6980a1569c46" containerName="main"
Sep 10 21:43:38.105298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105263    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2594c81-0d32-448a-bcbe-2d1b7d1b3690" containerName="payload-pre-processing"
Sep 10 21:43:38.105298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105272    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2594c81-0d32-448a-bcbe-2d1b7d1b3690" containerName="payload-pre-processing"
Sep 10 21:43:38.105298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105293    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8693d567-a118-4ce3-a2f8-4c13d7f9b197" containerName="payload-processing"
Sep 10 21:43:38.105641 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105302    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8693d567-a118-4ce3-a2f8-4c13d7f9b197" containerName="payload-processing"
Sep 10 21:43:38.105641 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105391    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c1e697-6baf-4491-bb81-48b1a96d3314" containerName="main"
Sep 10 21:43:38.105641 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105403    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2594c81-0d32-448a-bcbe-2d1b7d1b3690" containerName="payload-pre-processing"
Sep 10 21:43:38.105641 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105442    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff9edc6-532e-4286-bfd4-6980a1569c46" containerName="main"
Sep 10 21:43:38.105641 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.105454    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8693d567-a118-4ce3-a2f8-4c13d7f9b197" containerName="payload-processing"
Sep 10 21:43:38.109750 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.109728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.112802 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.112778    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"kube-root-ca.crt\""
Sep 10 21:43:38.112906 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.112783    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"rate-test-model-67d5e2-kserve-self-signed-certs\""
Sep 10 21:43:38.112906 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.112783    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"default-dockercfg-6kjsn\""
Sep 10 21:43:38.113018 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.112809    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67d5e2\"/\"openshift-service-ca.crt\""
Sep 10 21:43:38.117406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.117383    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"]
Sep 10 21:43:38.262843 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.262804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e919c883-1d01-4b7e-92c1-74dff7b851eb-tls-certs\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.262843 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.262846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-dshm\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.263058 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.262912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-tmp-dir\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.263058 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.262991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-model-cache\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.263058 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.263030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-home\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.263058 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.263046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf2hq\" (UniqueName: \"kubernetes.io/projected/e919c883-1d01-4b7e-92c1-74dff7b851eb-kube-api-access-nf2hq\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.363915 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.363825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-model-cache\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.363915 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.363888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-home\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.363915 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.363912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf2hq\" (UniqueName: \"kubernetes.io/projected/e919c883-1d01-4b7e-92c1-74dff7b851eb-kube-api-access-nf2hq\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.364157 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.363935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e919c883-1d01-4b7e-92c1-74dff7b851eb-tls-certs\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.364157 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.363952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-dshm\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.364157 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.364004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-tmp-dir\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.364276 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.364249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-model-cache\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.364315 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.364300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-home\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.364401 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.364377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-tmp-dir\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.366379 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.366350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-dshm\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.366644 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.366627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e919c883-1d01-4b7e-92c1-74dff7b851eb-tls-certs\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.371752 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.371729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf2hq\" (UniqueName: \"kubernetes.io/projected/e919c883-1d01-4b7e-92c1-74dff7b851eb-kube-api-access-nf2hq\") pod \"rate-test-model-67d5e2-kserve-577799fbc-ndkvg\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") " pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.421579 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.421552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:38.695738 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.695711    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"]
Sep 10 21:43:38.698286 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:43:38.698257    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode919c883_1d01_4b7e_92c1_74dff7b851eb.slice/crio-0fc4848ae85c0359b61116fc8fa4bebe60be7425bfe76602d94ffacb7211acd3 WatchSource:0}: Error finding container 0fc4848ae85c0359b61116fc8fa4bebe60be7425bfe76602d94ffacb7211acd3: Status 404 returned error can't find the container with id 0fc4848ae85c0359b61116fc8fa4bebe60be7425bfe76602d94ffacb7211acd3
Sep 10 21:43:38.928502 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.928463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg" event={"ID":"e919c883-1d01-4b7e-92c1-74dff7b851eb","Type":"ContainerStarted","Data":"773e6b7051132097d499c693e47bb136f8aacb09c85c7521d11b687c3b008ad5"}
Sep 10 21:43:38.928502 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.928504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg" event={"ID":"e919c883-1d01-4b7e-92c1-74dff7b851eb","Type":"ContainerStarted","Data":"0fc4848ae85c0359b61116fc8fa4bebe60be7425bfe76602d94ffacb7211acd3"}
Sep 10 21:43:38.948248 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:38.948144    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg" podStartSLOduration=0.948128241 podStartE2EDuration="948.128241ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:43:38.947649912 +0000 UTC m=+1457.595564081" watchObservedRunningTime="2026-09-10 21:43:38.948128241 +0000 UTC m=+1457.596042451"
Sep 10 21:43:39.992024 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:39.991982    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"]
Sep 10 21:43:39.992534 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:39.992222    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9" podUID="d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda" containerName="payload-pre-processing" containerID="cri-o://ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b" gracePeriod=30
Sep 10 21:43:40.239622 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.239595    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:43:40.383885 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.383797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwg2n\" (UniqueName: \"kubernetes.io/projected/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-kube-api-access-dwg2n\") pod \"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda\" (UID: \"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda\") "
Sep 10 21:43:40.384040 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.383905    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-plugins-config-volume\") pod \"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda\" (UID: \"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda\") "
Sep 10 21:43:40.384320 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.384296    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda" (UID: "d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:43:40.386151 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.386128    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-kube-api-access-dwg2n" (OuterVolumeSpecName: "kube-api-access-dwg2n") pod "d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda" (UID: "d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda"). InnerVolumeSpecName "kube-api-access-dwg2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:43:40.485038 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.484996    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwg2n\" (UniqueName: \"kubernetes.io/projected/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-kube-api-access-dwg2n\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:40.485038 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.485032    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:40.940891 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.940851    2580 generic.go:356] "Generic (PLEG): container finished" podID="d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda" containerID="ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b" exitCode=0
Sep 10 21:43:40.940891 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.940898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9" event={"ID":"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda","Type":"ContainerDied","Data":"ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b"}
Sep 10 21:43:40.941266 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.940901    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"
Sep 10 21:43:40.941266 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.940948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9" event={"ID":"d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda","Type":"ContainerDied","Data":"bc203a903e84fbb3ce077bbbee434bfbd6208ed7754d8851457180bb8959ef0c"}
Sep 10 21:43:40.941266 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.940973    2580 scope.go:117] "RemoveContainer" containerID="ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b"
Sep 10 21:43:40.950463 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.950444    2580 scope.go:117] "RemoveContainer" containerID="ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b"
Sep 10 21:43:40.950803 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:43:40.950785    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b\": container with ID starting with ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b not found: ID does not exist" containerID="ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b"
Sep 10 21:43:40.950873 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.950815    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b"} err="failed to get container status \"ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b\": rpc error: code = NotFound desc = could not find container \"ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b\": container with ID starting with ec13d192e2a30f196573c9c542213e77c6c0f7328708cd217b18a9ddaf74322b not found: ID does not exist"
Sep 10 21:43:40.963316 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.963283    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"]
Sep 10 21:43:40.967489 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:40.967460    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a867da-9cf764c86-728l9"]
Sep 10 21:43:41.820451 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:41.820395    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda" path="/var/lib/kubelet/pods/d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda/volumes"
Sep 10 21:43:42.026872 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:42.026837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:43:48.422369 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:48.422332    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:48.422789 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:48.422636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:48.435518 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:48.435495    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:48.993707 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:48.993675    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:43:57.802473 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:57.802435    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"]
Sep 10 21:43:57.803046 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:57.802709    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw" podUID="ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1" containerName="payload-processing" containerID="cri-o://504e101b81165b5bb723dd0e6457b6027431be309200382b729a10415a1b310d" gracePeriod=30
Sep 10 21:43:57.808389 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:57.808359    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"]
Sep 10 21:43:57.808676 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:57.808597    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv" podUID="69e231f2-2929-437a-be9c-134228a2719c" containerName="payload-pre-processing" containerID="cri-o://8ad768c404609f1ac7ff9bb5c92ea9b43d4a362973e854452af34838a7682dcb" gracePeriod=30
Sep 10 21:43:58.034581 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.034541    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1" containerID="504e101b81165b5bb723dd0e6457b6027431be309200382b729a10415a1b310d" exitCode=0
Sep 10 21:43:58.034746 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.034611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw" event={"ID":"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1","Type":"ContainerDied","Data":"504e101b81165b5bb723dd0e6457b6027431be309200382b729a10415a1b310d"}
Sep 10 21:43:58.036981 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.036954    2580 generic.go:356] "Generic (PLEG): container finished" podID="69e231f2-2929-437a-be9c-134228a2719c" containerID="8ad768c404609f1ac7ff9bb5c92ea9b43d4a362973e854452af34838a7682dcb" exitCode=0
Sep 10 21:43:58.037126 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.036991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv" event={"ID":"69e231f2-2929-437a-be9c-134228a2719c","Type":"ContainerDied","Data":"8ad768c404609f1ac7ff9bb5c92ea9b43d4a362973e854452af34838a7682dcb"}
Sep 10 21:43:58.068772 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.068706    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:43:58.101806 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.101778    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:43:58.158152 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.158116    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95mkd\" (UniqueName: \"kubernetes.io/projected/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-kube-api-access-95mkd\") pod \"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1\" (UID: \"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1\") "
Sep 10 21:43:58.158329 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.158179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-plugins-config-volume\") pod \"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1\" (UID: \"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1\") "
Sep 10 21:43:58.158329 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.158227    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e231f2-2929-437a-be9c-134228a2719c-plugins-config-volume\") pod \"69e231f2-2929-437a-be9c-134228a2719c\" (UID: \"69e231f2-2929-437a-be9c-134228a2719c\") "
Sep 10 21:43:58.158329 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.158264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpgfr\" (UniqueName: \"kubernetes.io/projected/69e231f2-2929-437a-be9c-134228a2719c-kube-api-access-qpgfr\") pod \"69e231f2-2929-437a-be9c-134228a2719c\" (UID: \"69e231f2-2929-437a-be9c-134228a2719c\") "
Sep 10 21:43:58.158699 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.158673    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e231f2-2929-437a-be9c-134228a2719c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69e231f2-2929-437a-be9c-134228a2719c" (UID: "69e231f2-2929-437a-be9c-134228a2719c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:43:58.158815 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.158672    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1" (UID: "ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:43:58.160453 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.160403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-kube-api-access-95mkd" (OuterVolumeSpecName: "kube-api-access-95mkd") pod "ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1" (UID: "ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1"). InnerVolumeSpecName "kube-api-access-95mkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:43:58.160966 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.160940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e231f2-2929-437a-be9c-134228a2719c-kube-api-access-qpgfr" (OuterVolumeSpecName: "kube-api-access-qpgfr") pod "69e231f2-2929-437a-be9c-134228a2719c" (UID: "69e231f2-2929-437a-be9c-134228a2719c"). InnerVolumeSpecName "kube-api-access-qpgfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:43:58.259256 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.259216    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e231f2-2929-437a-be9c-134228a2719c-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:58.259256 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.259250    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpgfr\" (UniqueName: \"kubernetes.io/projected/69e231f2-2929-437a-be9c-134228a2719c-kube-api-access-qpgfr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:58.259256 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.259260    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95mkd\" (UniqueName: \"kubernetes.io/projected/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-kube-api-access-95mkd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:58.259537 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:58.259268    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:43:59.043020 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.042989    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"
Sep 10 21:43:59.043487 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.042996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw" event={"ID":"ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1","Type":"ContainerDied","Data":"da30d86f43f103f1c36c6144a441fb21073dff5c56d1af2fbebeed673a100af2"}
Sep 10 21:43:59.043487 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.043125    2580 scope.go:117] "RemoveContainer" containerID="504e101b81165b5bb723dd0e6457b6027431be309200382b729a10415a1b310d"
Sep 10 21:43:59.044567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.044533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv" event={"ID":"69e231f2-2929-437a-be9c-134228a2719c","Type":"ContainerDied","Data":"a6ec4507693e5ce846fae15fa3a1e6b01f424f31fc9d4eeee5f4d132081720b4"}
Sep 10 21:43:59.044567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.044554    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"
Sep 10 21:43:59.053313 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.053296    2580 scope.go:117] "RemoveContainer" containerID="8ad768c404609f1ac7ff9bb5c92ea9b43d4a362973e854452af34838a7682dcb"
Sep 10 21:43:59.067730 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.067706    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"]
Sep 10 21:43:59.075209 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.075161    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4b45c2-7488cff5b5-qn5hw"]
Sep 10 21:43:59.085535 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.085514    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"]
Sep 10 21:43:59.091286 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.091264    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4b45c2-566776d694-f6xzv"]
Sep 10 21:43:59.208617 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.208591    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:43:59.823780 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.823743    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69e231f2-2929-437a-be9c-134228a2719c" path="/var/lib/kubelet/pods/69e231f2-2929-437a-be9c-134228a2719c/volumes"
Sep 10 21:43:59.824145 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:43:59.824131    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1" path="/var/lib/kubelet/pods/ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1/volumes"
Sep 10 21:44:09.910649 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:09.910608    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:44:22.774724 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:22.774689    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"]
Sep 10 21:44:22.775146 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:22.774925    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy" containerID="cri-o://be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4" gracePeriod=30
Sep 10 21:44:22.930257 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:22.930217    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:44:27.155106 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:27.155074    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"]
Sep 10 21:44:28.028218 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.028191    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:44:28.054248 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.053899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e0964b-d51f-44e3-83d4-546d9e7e7718-istiod-ca-cert\") pod \"19e0964b-d51f-44e3-83d4-546d9e7e7718\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") "
Sep 10 21:44:28.054248 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.053960    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-credential-socket\") pod \"19e0964b-d51f-44e3-83d4-546d9e7e7718\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") "
Sep 10 21:44:28.054248 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-envoy\") pod \"19e0964b-d51f-44e3-83d4-546d9e7e7718\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") "
Sep 10 21:44:28.054248 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-token\") pod \"19e0964b-d51f-44e3-83d4-546d9e7e7718\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") "
Sep 10 21:44:28.054248 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054112    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-socket\") pod \"19e0964b-d51f-44e3-83d4-546d9e7e7718\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") "
Sep 10 21:44:28.054248 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqwfv\" (UniqueName: \"kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-kube-api-access-jqwfv\") pod \"19e0964b-d51f-44e3-83d4-546d9e7e7718\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") "
Sep 10 21:44:28.054248 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054208    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-podinfo\") pod \"19e0964b-d51f-44e3-83d4-546d9e7e7718\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") "
Sep 10 21:44:28.054248 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054225    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-data\") pod \"19e0964b-d51f-44e3-83d4-546d9e7e7718\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") "
Sep 10 21:44:28.054865 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054261    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-certs\") pod \"19e0964b-d51f-44e3-83d4-546d9e7e7718\" (UID: \"19e0964b-d51f-44e3-83d4-546d9e7e7718\") "
Sep 10 21:44:28.054865 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054272    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e0964b-d51f-44e3-83d4-546d9e7e7718-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19e0964b-d51f-44e3-83d4-546d9e7e7718" (UID: "19e0964b-d51f-44e3-83d4-546d9e7e7718"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:44:28.054865 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19e0964b-d51f-44e3-83d4-546d9e7e7718" (UID: "19e0964b-d51f-44e3-83d4-546d9e7e7718"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:28.054865 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054590    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e0964b-d51f-44e3-83d4-546d9e7e7718-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:28.054865 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054677    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-data" (OuterVolumeSpecName: "istio-data") pod "19e0964b-d51f-44e3-83d4-546d9e7e7718" (UID: "19e0964b-d51f-44e3-83d4-546d9e7e7718"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:28.054865 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19e0964b-d51f-44e3-83d4-546d9e7e7718" (UID: "19e0964b-d51f-44e3-83d4-546d9e7e7718"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:28.055196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.054883    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19e0964b-d51f-44e3-83d4-546d9e7e7718" (UID: "19e0964b-d51f-44e3-83d4-546d9e7e7718"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:28.056778 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.056749    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19e0964b-d51f-44e3-83d4-546d9e7e7718" (UID: "19e0964b-d51f-44e3-83d4-546d9e7e7718"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:44:28.057066 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.057032    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-kube-api-access-jqwfv" (OuterVolumeSpecName: "kube-api-access-jqwfv") pod "19e0964b-d51f-44e3-83d4-546d9e7e7718" (UID: "19e0964b-d51f-44e3-83d4-546d9e7e7718"). InnerVolumeSpecName "kube-api-access-jqwfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:44:28.057226 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.057173    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19e0964b-d51f-44e3-83d4-546d9e7e7718" (UID: "19e0964b-d51f-44e3-83d4-546d9e7e7718"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:28.057430 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.057383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-token" (OuterVolumeSpecName: "istio-token") pod "19e0964b-d51f-44e3-83d4-546d9e7e7718" (UID: "19e0964b-d51f-44e3-83d4-546d9e7e7718"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:44:28.097342 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.097309    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"]
Sep 10 21:44:28.155850 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.155813    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:28.155850 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.155844    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:28.155850 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.155854    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:28.156320 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.155864    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqwfv\" (UniqueName: \"kubernetes.io/projected/19e0964b-d51f-44e3-83d4-546d9e7e7718-kube-api-access-jqwfv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:28.156320 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.155874    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:28.156320 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.155882    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:28.156320 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.155891    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:28.156320 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.155898    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e0964b-d51f-44e3-83d4-546d9e7e7718-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:28.195676 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.195641    2580 generic.go:356] "Generic (PLEG): container finished" podID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerID="be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4" exitCode=0
Sep 10 21:44:28.195868 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.195717    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"
Sep 10 21:44:28.195868 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.195716    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" event={"ID":"19e0964b-d51f-44e3-83d4-546d9e7e7718","Type":"ContainerDied","Data":"be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4"}
Sep 10 21:44:28.195868 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.195863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t" event={"ID":"19e0964b-d51f-44e3-83d4-546d9e7e7718","Type":"ContainerDied","Data":"bf15faa13c6742391158658e08cc7a4dfcab6f06d623b17f7f2ed45719efccc0"}
Sep 10 21:44:28.196028 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.195885    2580 scope.go:117] "RemoveContainer" containerID="be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4"
Sep 10 21:44:28.205743 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.205729    2580 scope.go:117] "RemoveContainer" containerID="1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264"
Sep 10 21:44:28.213688 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.213672    2580 scope.go:117] "RemoveContainer" containerID="be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4"
Sep 10 21:44:28.213974 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:44:28.213951    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4\": container with ID starting with be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4 not found: ID does not exist" containerID="be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4"
Sep 10 21:44:28.214019 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.213984    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4"} err="failed to get container status \"be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4\": rpc error: code = NotFound desc = could not find container \"be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4\": container with ID starting with be1194081bdf0afba177505153263e7dac2604f827c9f454e28d355a37d9ebd4 not found: ID does not exist"
Sep 10 21:44:28.214019 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.214002    2580 scope.go:117] "RemoveContainer" containerID="1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264"
Sep 10 21:44:28.214258 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:44:28.214238    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264\": container with ID starting with 1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264 not found: ID does not exist" containerID="1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264"
Sep 10 21:44:28.214322 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.214267    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264"} err="failed to get container status \"1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264\": rpc error: code = NotFound desc = could not find container \"1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264\": container with ID starting with 1c5e0dd2d4826dca1c1395ddc8410ec3651e1ca06ae3d72b870e42faf8b3b264 not found: ID does not exist"
Sep 10 21:44:28.218588 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.218543    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"]
Sep 10 21:44:28.220674 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:28.220656    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b4c9bb-openshift-default-7584cd9bcb-sbs5t"]
Sep 10 21:44:29.821210 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:29.821166    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" path="/var/lib/kubelet/pods/19e0964b-d51f-44e3-83d4-546d9e7e7718/volumes"
Sep 10 21:44:32.218454 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:32.218406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:44:33.636098 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636056    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"]
Sep 10 21:44:33.636644 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636625    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e231f2-2929-437a-be9c-134228a2719c" containerName="payload-pre-processing"
Sep 10 21:44:33.636723 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636648    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e231f2-2929-437a-be9c-134228a2719c" containerName="payload-pre-processing"
Sep 10 21:44:33.636723 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636661    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda" containerName="payload-pre-processing"
Sep 10 21:44:33.636723 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636669    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda" containerName="payload-pre-processing"
Sep 10 21:44:33.636723 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636702    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1" containerName="payload-processing"
Sep 10 21:44:33.636723 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636711    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1" containerName="payload-processing"
Sep 10 21:44:33.636723 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636722    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy"
Sep 10 21:44:33.637024 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636731    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy"
Sep 10 21:44:33.637024 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636827    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e231f2-2929-437a-be9c-134228a2719c" containerName="payload-pre-processing"
Sep 10 21:44:33.637024 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636845    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy"
Sep 10 21:44:33.637024 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636854    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy"
Sep 10 21:44:33.637024 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636865    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad6f83f9-f3c2-4cda-abdc-5bc7c2f69bf1" containerName="payload-processing"
Sep 10 21:44:33.637024 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.636874    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2187fe6-00cd-4cc4-ad3a-4d1c2219fcda" containerName="payload-pre-processing"
Sep 10 21:44:33.641722 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.641699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:44:33.644037 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.644017    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8291ef\""
Sep 10 21:44:33.644163 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.644071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8291ef-dockercfg-rgmdm\""
Sep 10 21:44:33.648285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.648258    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"]
Sep 10 21:44:33.675635 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.675604    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"]
Sep 10 21:44:33.675964 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.675952    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy"
Sep 10 21:44:33.676008 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.675966    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e0964b-d51f-44e3-83d4-546d9e7e7718" containerName="istio-proxy"
Sep 10 21:44:33.678992 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.678976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:44:33.693605 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.693580    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"]
Sep 10 21:44:33.709005 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.708976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e65351a-7acb-4339-8e46-7d32316f6c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4\" (UID: \"8e65351a-7acb-4339-8e46-7d32316f6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:44:33.709134 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.709108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7bjc\" (UniqueName: \"kubernetes.io/projected/8e65351a-7acb-4339-8e46-7d32316f6c86-kube-api-access-c7bjc\") pod \"payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4\" (UID: \"8e65351a-7acb-4339-8e46-7d32316f6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:44:33.810011 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.809974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e65351a-7acb-4339-8e46-7d32316f6c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4\" (UID: \"8e65351a-7acb-4339-8e46-7d32316f6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:44:33.810186 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.810049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7bjc\" (UniqueName: \"kubernetes.io/projected/8e65351a-7acb-4339-8e46-7d32316f6c86-kube-api-access-c7bjc\") pod \"payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4\" (UID: \"8e65351a-7acb-4339-8e46-7d32316f6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:44:33.810186 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.810098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpfqx\" (UniqueName: \"kubernetes.io/projected/2dd648bb-3025-40be-b056-f171ff9ccd84-kube-api-access-wpfqx\") pod \"payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz\" (UID: \"2dd648bb-3025-40be-b056-f171ff9ccd84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:44:33.810186 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.810145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd648bb-3025-40be-b056-f171ff9ccd84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz\" (UID: \"2dd648bb-3025-40be-b056-f171ff9ccd84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:44:33.810732 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.810713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e65351a-7acb-4339-8e46-7d32316f6c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4\" (UID: \"8e65351a-7acb-4339-8e46-7d32316f6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:44:33.823957 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.818944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7bjc\" (UniqueName: \"kubernetes.io/projected/8e65351a-7acb-4339-8e46-7d32316f6c86-kube-api-access-c7bjc\") pod \"payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4\" (UID: \"8e65351a-7acb-4339-8e46-7d32316f6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:44:33.910995 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.910908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpfqx\" (UniqueName: \"kubernetes.io/projected/2dd648bb-3025-40be-b056-f171ff9ccd84-kube-api-access-wpfqx\") pod \"payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz\" (UID: \"2dd648bb-3025-40be-b056-f171ff9ccd84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:44:33.910995 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.910950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd648bb-3025-40be-b056-f171ff9ccd84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz\" (UID: \"2dd648bb-3025-40be-b056-f171ff9ccd84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:44:33.911608 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.911589    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd648bb-3025-40be-b056-f171ff9ccd84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz\" (UID: \"2dd648bb-3025-40be-b056-f171ff9ccd84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:44:33.918917 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.918888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpfqx\" (UniqueName: \"kubernetes.io/projected/2dd648bb-3025-40be-b056-f171ff9ccd84-kube-api-access-wpfqx\") pod \"payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz\" (UID: \"2dd648bb-3025-40be-b056-f171ff9ccd84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:44:33.953692 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.953668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:44:33.988474 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:33.988406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:44:34.241173 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:34.241151    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"]
Sep 10 21:44:34.243442 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:44:34.243395    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e65351a_7acb_4339_8e46_7d32316f6c86.slice/crio-fbb2f013810399f531c2b16d8be72aae40c83d3e53756514f95df3ddd61bced6 WatchSource:0}: Error finding container fbb2f013810399f531c2b16d8be72aae40c83d3e53756514f95df3ddd61bced6: Status 404 returned error can't find the container with id fbb2f013810399f531c2b16d8be72aae40c83d3e53756514f95df3ddd61bced6
Sep 10 21:44:34.272321 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:34.272279    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"]
Sep 10 21:44:34.276516 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:44:34.276493    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd648bb_3025_40be_b056_f171ff9ccd84.slice/crio-6353bffae3f50b0410d612f37f1561ddd1922cfd5d771e35491da67ef02603cb WatchSource:0}: Error finding container 6353bffae3f50b0410d612f37f1561ddd1922cfd5d771e35491da67ef02603cb: Status 404 returned error can't find the container with id 6353bffae3f50b0410d612f37f1561ddd1922cfd5d771e35491da67ef02603cb
Sep 10 21:44:35.232484 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:35.232447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz" event={"ID":"2dd648bb-3025-40be-b056-f171ff9ccd84","Type":"ContainerStarted","Data":"69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa"}
Sep 10 21:44:35.232948 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:35.232518    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:44:35.232948 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:35.232543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz" event={"ID":"2dd648bb-3025-40be-b056-f171ff9ccd84","Type":"ContainerStarted","Data":"6353bffae3f50b0410d612f37f1561ddd1922cfd5d771e35491da67ef02603cb"}
Sep 10 21:44:35.234191 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:35.234164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4" event={"ID":"8e65351a-7acb-4339-8e46-7d32316f6c86","Type":"ContainerStarted","Data":"a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627"}
Sep 10 21:44:35.234191 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:35.234205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4" event={"ID":"8e65351a-7acb-4339-8e46-7d32316f6c86","Type":"ContainerStarted","Data":"fbb2f013810399f531c2b16d8be72aae40c83d3e53756514f95df3ddd61bced6"}
Sep 10 21:44:35.234373 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:35.234239    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:44:35.250263 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:35.250220    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz" podStartSLOduration=2.250202203 podStartE2EDuration="2.250202203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:44:35.247900451 +0000 UTC m=+1513.895814660" watchObservedRunningTime="2026-09-10 21:44:35.250202203 +0000 UTC m=+1513.898116368"
Sep 10 21:44:35.264544 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:35.264497    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4" podStartSLOduration=2.264482168 podStartE2EDuration="2.264482168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:44:35.262725717 +0000 UTC m=+1513.910639886" watchObservedRunningTime="2026-09-10 21:44:35.264482168 +0000 UTC m=+1513.912396337"
Sep 10 21:44:42.175689 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.175634    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n" podUID="e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" containerName="main" containerID="cri-o://20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511" gracePeriod=45
Sep 10 21:44:42.213895 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.213858    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:44:42.427687 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.427625    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:44:42.492514 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.492484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-dshm\") pod \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") "
Sep 10 21:44:42.492677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.492530    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tls-certs\") pod \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") "
Sep 10 21:44:42.492677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.492577    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-model-cache\") pod \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") "
Sep 10 21:44:42.492677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.492642    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptmjx\" (UniqueName: \"kubernetes.io/projected/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-kube-api-access-ptmjx\") pod \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") "
Sep 10 21:44:42.492677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.492657    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-home\") pod \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") "
Sep 10 21:44:42.492891 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.492697    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tmp-dir\") pod \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\" (UID: \"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173\") "
Sep 10 21:44:42.492960 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.492891    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" (UID: "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:42.492960 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.492934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-home" (OuterVolumeSpecName: "home") pod "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" (UID: "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:42.493055 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.493020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" (UID: "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:42.493055 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.493047    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:42.493164 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.493066    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:42.494863 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.494840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-dshm" (OuterVolumeSpecName: "dshm") pod "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" (UID: "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:42.494968 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.494870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" (UID: "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:44:42.494968 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.494922    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-kube-api-access-ptmjx" (OuterVolumeSpecName: "kube-api-access-ptmjx") pod "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" (UID: "e6a6a4bc-be9c-457d-95c1-a7d89d8eb173"). InnerVolumeSpecName "kube-api-access-ptmjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:44:42.594163 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.594120    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptmjx\" (UniqueName: \"kubernetes.io/projected/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-kube-api-access-ptmjx\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:42.594163 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.594155    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:42.594163 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.594165    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:42.594163 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:42.594173    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:43.117129 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.117023    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg" podUID="e919c883-1d01-4b7e-92c1-74dff7b851eb" containerName="main" containerID="cri-o://773e6b7051132097d499c693e47bb136f8aacb09c85c7521d11b687c3b008ad5" gracePeriod=45
Sep 10 21:44:43.281269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.281237    2580 generic.go:356] "Generic (PLEG): container finished" podID="e919c883-1d01-4b7e-92c1-74dff7b851eb" containerID="773e6b7051132097d499c693e47bb136f8aacb09c85c7521d11b687c3b008ad5" exitCode=0
Sep 10 21:44:43.281680 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.281315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg" event={"ID":"e919c883-1d01-4b7e-92c1-74dff7b851eb","Type":"ContainerDied","Data":"773e6b7051132097d499c693e47bb136f8aacb09c85c7521d11b687c3b008ad5"}
Sep 10 21:44:43.282821 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.282792    2580 generic.go:356] "Generic (PLEG): container finished" podID="e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" containerID="20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511" exitCode=0
Sep 10 21:44:43.282956 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.282833    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"
Sep 10 21:44:43.282956 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.282853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n" event={"ID":"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173","Type":"ContainerDied","Data":"20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511"}
Sep 10 21:44:43.282956 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.282890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n" event={"ID":"e6a6a4bc-be9c-457d-95c1-a7d89d8eb173","Type":"ContainerDied","Data":"8f0287c04d3022b77b3e42c0c061a6d9a74a28c0c3aff7f74a749ae478d78cc4"}
Sep 10 21:44:43.282956 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.282908    2580 scope.go:117] "RemoveContainer" containerID="20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511"
Sep 10 21:44:43.293096 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.293082    2580 scope.go:117] "RemoveContainer" containerID="20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511"
Sep 10 21:44:43.293375 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:44:43.293356    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511\": container with ID starting with 20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511 not found: ID does not exist" containerID="20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511"
Sep 10 21:44:43.293492 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.293383    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511"} err="failed to get container status \"20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511\": rpc error: code = NotFound desc = could not find container \"20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511\": container with ID starting with 20d80740c2f519d48490bad8ae471942638a8ea5aca80add2639d13a952b2511 not found: ID does not exist"
Sep 10 21:44:43.311436 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.311386    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"]
Sep 10 21:44:43.313137 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.313115    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2d9e68/rate-test-model-2d9e68-kserve-7999bb9d4c-5mf8n"]
Sep 10 21:44:43.380854 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.380791    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:44:43.502021 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.501991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf2hq\" (UniqueName: \"kubernetes.io/projected/e919c883-1d01-4b7e-92c1-74dff7b851eb-kube-api-access-nf2hq\") pod \"e919c883-1d01-4b7e-92c1-74dff7b851eb\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") "
Sep 10 21:44:43.502021 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.502028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-model-cache\") pod \"e919c883-1d01-4b7e-92c1-74dff7b851eb\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") "
Sep 10 21:44:43.502269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.502055    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-home\") pod \"e919c883-1d01-4b7e-92c1-74dff7b851eb\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") "
Sep 10 21:44:43.502269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.502131    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e919c883-1d01-4b7e-92c1-74dff7b851eb-tls-certs\") pod \"e919c883-1d01-4b7e-92c1-74dff7b851eb\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") "
Sep 10 21:44:43.502269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.502149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-tmp-dir\") pod \"e919c883-1d01-4b7e-92c1-74dff7b851eb\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") "
Sep 10 21:44:43.502269 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.502186    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-dshm\") pod \"e919c883-1d01-4b7e-92c1-74dff7b851eb\" (UID: \"e919c883-1d01-4b7e-92c1-74dff7b851eb\") "
Sep 10 21:44:43.502501 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.502265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "e919c883-1d01-4b7e-92c1-74dff7b851eb" (UID: "e919c883-1d01-4b7e-92c1-74dff7b851eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:43.502501 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.502318    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-home" (OuterVolumeSpecName: "home") pod "e919c883-1d01-4b7e-92c1-74dff7b851eb" (UID: "e919c883-1d01-4b7e-92c1-74dff7b851eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:43.502501 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.502460    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e919c883-1d01-4b7e-92c1-74dff7b851eb" (UID: "e919c883-1d01-4b7e-92c1-74dff7b851eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:43.502501 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.502466    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:43.502678 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.502507    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:43.504379 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.504347    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e919c883-1d01-4b7e-92c1-74dff7b851eb-kube-api-access-nf2hq" (OuterVolumeSpecName: "kube-api-access-nf2hq") pod "e919c883-1d01-4b7e-92c1-74dff7b851eb" (UID: "e919c883-1d01-4b7e-92c1-74dff7b851eb"). InnerVolumeSpecName "kube-api-access-nf2hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:44:43.504379 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.504373    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e919c883-1d01-4b7e-92c1-74dff7b851eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e919c883-1d01-4b7e-92c1-74dff7b851eb" (UID: "e919c883-1d01-4b7e-92c1-74dff7b851eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:44:43.504543 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.504378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-dshm" (OuterVolumeSpecName: "dshm") pod "e919c883-1d01-4b7e-92c1-74dff7b851eb" (UID: "e919c883-1d01-4b7e-92c1-74dff7b851eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:44:43.603929 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.603892    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e919c883-1d01-4b7e-92c1-74dff7b851eb-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:43.603929 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.603923    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:43.603929 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.603936    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e919c883-1d01-4b7e-92c1-74dff7b851eb-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:43.604151 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.603947    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf2hq\" (UniqueName: \"kubernetes.io/projected/e919c883-1d01-4b7e-92c1-74dff7b851eb-kube-api-access-nf2hq\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:44:43.820827 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:43.820792    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" path="/var/lib/kubelet/pods/e6a6a4bc-be9c-457d-95c1-a7d89d8eb173/volumes"
Sep 10 21:44:44.289462 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:44.289435    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"
Sep 10 21:44:44.289933 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:44.289428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg" event={"ID":"e919c883-1d01-4b7e-92c1-74dff7b851eb","Type":"ContainerDied","Data":"0fc4848ae85c0359b61116fc8fa4bebe60be7425bfe76602d94ffacb7211acd3"}
Sep 10 21:44:44.289933 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:44.289558    2580 scope.go:117] "RemoveContainer" containerID="773e6b7051132097d499c693e47bb136f8aacb09c85c7521d11b687c3b008ad5"
Sep 10 21:44:44.305013 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:44.304988    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"]
Sep 10 21:44:44.308709 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:44.308685    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67d5e2/rate-test-model-67d5e2-kserve-577799fbc-ndkvg"]
Sep 10 21:44:45.820055 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:45.820019    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e919c883-1d01-4b7e-92c1-74dff7b851eb" path="/var/lib/kubelet/pods/e919c883-1d01-4b7e-92c1-74dff7b851eb/volumes"
Sep 10 21:44:46.240537 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:46.240507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:44:46.240701 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:46.240576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:44:57.596932 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.596895    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"]
Sep 10 21:44:57.597297 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.597275    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" containerName="main"
Sep 10 21:44:57.597297 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.597286    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" containerName="main"
Sep 10 21:44:57.597297 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.597296    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e919c883-1d01-4b7e-92c1-74dff7b851eb" containerName="main"
Sep 10 21:44:57.597448 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.597302    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e919c883-1d01-4b7e-92c1-74dff7b851eb" containerName="main"
Sep 10 21:44:57.597945 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.597916    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6a6a4bc-be9c-457d-95c1-a7d89d8eb173" containerName="main"
Sep 10 21:44:57.598140 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.598128    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e919c883-1d01-4b7e-92c1-74dff7b851eb" containerName="main"
Sep 10 21:44:57.602271 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.602247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.605992 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.605966    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2fbf53-openshift-default-dockercfg-w88sd\""
Sep 10 21:44:57.630156 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.630131    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"]
Sep 10 21:44:57.631326 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.631297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-envoy\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.631485 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.631343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-data\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.631485 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.631407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-token\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.631485 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.631457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-credential-socket\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.631485 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.631481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-podinfo\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.631752 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.631506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ade033-3cca-4b16-9cc1-558e416bd53b-istiod-ca-cert\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.631752 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.631559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-certs\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.631752 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.631584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-socket\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.631752 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.631640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrsqz\" (UniqueName: \"kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-kube-api-access-wrsqz\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.732994 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.732949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-token\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733200 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-credential-socket\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733200 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-podinfo\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733200 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ade033-3cca-4b16-9cc1-558e416bd53b-istiod-ca-cert\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733462 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-certs\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733462 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-socket\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733462 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrsqz\" (UniqueName: \"kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-kube-api-access-wrsqz\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733462 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-envoy\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733673 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-data\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733673 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-credential-socket\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733673 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-certs\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733823 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-socket\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733876 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-data\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.733954 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.733915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ade033-3cca-4b16-9cc1-558e416bd53b-istiod-ca-cert\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.736009 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.735976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-envoy\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.736197 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.736177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-podinfo\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.742730 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.742701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-token\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.746067 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.746041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrsqz\" (UniqueName: \"kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-kube-api-access-wrsqz\") pod \"e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") " pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:57.913531 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:57.913396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:58.190671 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:58.190642    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"]
Sep 10 21:44:58.193141 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:44:58.193118    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ade033_3cca_4b16_9cc1_558e416bd53b.slice/crio-3c19dc2ddcd578aed8c7f7ce08aa5dc6c21a82ae5b141d36aa894b991e3af4f6 WatchSource:0}: Error finding container 3c19dc2ddcd578aed8c7f7ce08aa5dc6c21a82ae5b141d36aa894b991e3af4f6: Status 404 returned error can't find the container with id 3c19dc2ddcd578aed8c7f7ce08aa5dc6c21a82ae5b141d36aa894b991e3af4f6
Sep 10 21:44:58.195187 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:58.195158    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:44:58.195244 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:58.195223    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:44:58.195286 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:58.195253    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:44:58.363341 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:58.363301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" event={"ID":"23ade033-3cca-4b16-9cc1-558e416bd53b","Type":"ContainerStarted","Data":"7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2"}
Sep 10 21:44:58.363530 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:58.363366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" event={"ID":"23ade033-3cca-4b16-9cc1-558e416bd53b","Type":"ContainerStarted","Data":"3c19dc2ddcd578aed8c7f7ce08aa5dc6c21a82ae5b141d36aa894b991e3af4f6"}
Sep 10 21:44:58.419282 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:58.419223    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" podStartSLOduration=1.41920588 podStartE2EDuration="1.41920588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:44:58.416321632 +0000 UTC m=+1537.064235799" watchObservedRunningTime="2026-09-10 21:44:58.41920588 +0000 UTC m=+1537.067120105"
Sep 10 21:44:58.914407 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:58.914373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:44:59.919112 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:44:59.919084    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:45:00.145856 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:00.145820    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4"]
Sep 10 21:45:00.149559 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:00.149534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4"
Sep 10 21:45:00.151891 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:00.151867    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wg7l\""
Sep 10 21:45:00.159890 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:00.159863    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4"]
Sep 10 21:45:00.162222 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:00.162202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4"
Sep 10 21:45:00.375527 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:00.375495    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:45:00.376660 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:00.376637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:45:00.441391 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:00.441363    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4"]
Sep 10 21:45:00.443355 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:45:00.443329    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc471c2ba_5c4e_45a5_991a_7c458bfc818d.slice/crio-9b3ffd8dae6c3d1f168dcea89d93c7b858e3ef6230cc8643547dc4054469cc68 WatchSource:0}: Error finding container 9b3ffd8dae6c3d1f168dcea89d93c7b858e3ef6230cc8643547dc4054469cc68: Status 404 returned error can't find the container with id 9b3ffd8dae6c3d1f168dcea89d93c7b858e3ef6230cc8643547dc4054469cc68
Sep 10 21:45:01.381867 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:01.381835    2580 generic.go:356] "Generic (PLEG): container finished" podID="c471c2ba-5c4e-45a5-991a-7c458bfc818d" containerID="46c9df030dc47e8e89187f8501d7ff84c4e1a53bbca5b774b74a3850bc1954c8" exitCode=0
Sep 10 21:45:01.382215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:01.381909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4" event={"ID":"c471c2ba-5c4e-45a5-991a-7c458bfc818d","Type":"ContainerDied","Data":"46c9df030dc47e8e89187f8501d7ff84c4e1a53bbca5b774b74a3850bc1954c8"}
Sep 10 21:45:01.382215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:01.381961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4" event={"ID":"c471c2ba-5c4e-45a5-991a-7c458bfc818d","Type":"ContainerStarted","Data":"9b3ffd8dae6c3d1f168dcea89d93c7b858e3ef6230cc8643547dc4054469cc68"}
Sep 10 21:45:02.129465 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.129429    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"]
Sep 10 21:45:02.132959 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.132936    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:45:02.135831 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.135801    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2fbf53-dockercfg-4rs9d\""
Sep 10 21:45:02.136136 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.136114    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2fbf53\""
Sep 10 21:45:02.144152 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.144129    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"]
Sep 10 21:45:02.174604 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.174571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tvxc\" (UniqueName: \"kubernetes.io/projected/ef3673cd-0191-4f84-9f90-2f949c21f95c-kube-api-access-5tvxc\") pod \"payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm\" (UID: \"ef3673cd-0191-4f84-9f90-2f949c21f95c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:45:02.174801 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.174737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3673cd-0191-4f84-9f90-2f949c21f95c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm\" (UID: \"ef3673cd-0191-4f84-9f90-2f949c21f95c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:45:02.275522 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.275485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tvxc\" (UniqueName: \"kubernetes.io/projected/ef3673cd-0191-4f84-9f90-2f949c21f95c-kube-api-access-5tvxc\") pod \"payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm\" (UID: \"ef3673cd-0191-4f84-9f90-2f949c21f95c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:45:02.275729 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.275572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3673cd-0191-4f84-9f90-2f949c21f95c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm\" (UID: \"ef3673cd-0191-4f84-9f90-2f949c21f95c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:45:02.276224 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.276206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3673cd-0191-4f84-9f90-2f949c21f95c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm\" (UID: \"ef3673cd-0191-4f84-9f90-2f949c21f95c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:45:02.283771 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.283745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tvxc\" (UniqueName: \"kubernetes.io/projected/ef3673cd-0191-4f84-9f90-2f949c21f95c-kube-api-access-5tvxc\") pod \"payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm\" (UID: \"ef3673cd-0191-4f84-9f90-2f949c21f95c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:45:02.444770 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.444742    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:45:02.522009 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.521981    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4"
Sep 10 21:45:02.718393 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:02.718354    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"]
Sep 10 21:45:02.720936 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:45:02.720898    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3673cd_0191_4f84_9f90_2f949c21f95c.slice/crio-1bef9983cf20ac7a3e6e37fcb2abad4a420c8720ee6ab34e8169f698c0732e61 WatchSource:0}: Error finding container 1bef9983cf20ac7a3e6e37fcb2abad4a420c8720ee6ab34e8169f698c0732e61: Status 404 returned error can't find the container with id 1bef9983cf20ac7a3e6e37fcb2abad4a420c8720ee6ab34e8169f698c0732e61
Sep 10 21:45:03.393810 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:03.393775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm" event={"ID":"ef3673cd-0191-4f84-9f90-2f949c21f95c","Type":"ContainerStarted","Data":"4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a"}
Sep 10 21:45:03.394030 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:03.393845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm" event={"ID":"ef3673cd-0191-4f84-9f90-2f949c21f95c","Type":"ContainerStarted","Data":"1bef9983cf20ac7a3e6e37fcb2abad4a420c8720ee6ab34e8169f698c0732e61"}
Sep 10 21:45:03.394030 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:03.393891    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:45:03.395108 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:03.395080    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4"
Sep 10 21:45:03.395235 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:03.395126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4" event={"ID":"c471c2ba-5c4e-45a5-991a-7c458bfc818d","Type":"ContainerDied","Data":"9b3ffd8dae6c3d1f168dcea89d93c7b858e3ef6230cc8643547dc4054469cc68"}
Sep 10 21:45:03.395235 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:03.395164    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b3ffd8dae6c3d1f168dcea89d93c7b858e3ef6230cc8643547dc4054469cc68"
Sep 10 21:45:03.414694 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:03.414647    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm" podStartSLOduration=1.414631551 podStartE2EDuration="1.414631551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:45:03.413613758 +0000 UTC m=+1542.061527927" watchObservedRunningTime="2026-09-10 21:45:03.414631551 +0000 UTC m=+1542.062545718"
Sep 10 21:45:05.813178 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:05.813144    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:45:14.403234 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:14.403197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:45:15.620808 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:15.619784    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": read tcp 10.133.0.2:56822->10.133.0.68:15021: read: connection reset by peer" start-of-body=
Sep 10 21:45:15.620808 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:15.619867    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": read tcp 10.133.0.2:56822->10.133.0.68:15021: read: connection reset by peer"
Sep 10 21:45:16.464173 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:16.464135    2580 generic.go:356] "Generic (PLEG): container finished" podID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerID="7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2" exitCode=0
Sep 10 21:45:16.464339 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:16.464208    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" event={"ID":"23ade033-3cca-4b16-9cc1-558e416bd53b","Type":"ContainerDied","Data":"7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2"}
Sep 10 21:45:16.464610 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:16.464596    2580 scope.go:117] "RemoveContainer" containerID="7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2"
Sep 10 21:45:16.468220 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:16.468188    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:45:16.468316 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:16.468301    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:45:16.468381 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:16.468354    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:45:17.475651 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:17.475615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" event={"ID":"23ade033-3cca-4b16-9cc1-558e416bd53b","Type":"ContainerStarted","Data":"821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253"}
Sep 10 21:45:17.914581 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:17.914470    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:45:17.914761 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:17.914744    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 10 21:45:17.914816 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:17.914794    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 10 21:45:18.914629 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:18.914586    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 10 21:45:18.915121 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:18.914651    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 10 21:45:19.914098 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:19.914058    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 10 21:45:19.914280 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:19.914119    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 10 21:45:20.914211 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:20.914159    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 10 21:45:20.914728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:20.914232    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 10 21:45:21.914455 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:21.914393    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 10 21:45:21.914841 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:21.914490    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 10 21:45:22.937244 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:22.937211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:45:22.937696 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:22.937490    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:45:22.938354 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:22.938336    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:45:30.525177 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:30.525147    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm_ef3673cd-0191-4f84-9f90-2f949c21f95c/payload-processing/0.log"
Sep 10 21:45:33.207215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.207182    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"]
Sep 10 21:45:33.207738 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.207585    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c471c2ba-5c4e-45a5-991a-7c458bfc818d" containerName="cleanup"
Sep 10 21:45:33.207738 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.207597    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c471c2ba-5c4e-45a5-991a-7c458bfc818d" containerName="cleanup"
Sep 10 21:45:33.207738 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.207695    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c471c2ba-5c4e-45a5-991a-7c458bfc818d" containerName="cleanup"
Sep 10 21:45:33.212407 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.212389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.215804 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.215777    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8291ef\"/\"default-dockercfg-88wrx\""
Sep 10 21:45:33.215929 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.215820    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8291ef\"/\"openshift-service-ca.crt\""
Sep 10 21:45:33.215929 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.215826    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8291ef\"/\"ipp-route-8291ef-kserve-self-signed-certs\""
Sep 10 21:45:33.215929 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.215865    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8291ef\"/\"kube-root-ca.crt\""
Sep 10 21:45:33.221321 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.221298    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"]
Sep 10 21:45:33.383267 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.383229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tmp-dir\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.383267 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.383267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-dshm\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.383507 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.383289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw2vm\" (UniqueName: \"kubernetes.io/projected/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-kube-api-access-rw2vm\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.383507 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.383357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tls-certs\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.383507 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.383431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-home\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.383622 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.383533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-model-cache\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.484299 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.484220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tmp-dir\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.484299 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.484251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-dshm\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.484299 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.484275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw2vm\" (UniqueName: \"kubernetes.io/projected/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-kube-api-access-rw2vm\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.484299 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.484297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tls-certs\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.484657 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.484339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-home\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.484657 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.484404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-model-cache\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.484657 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.484624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tmp-dir\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.484820 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.484768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-home\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.484820 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.484799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-model-cache\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.486662 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.486637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-dshm\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.486949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.486931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tls-certs\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.492296 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.492270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw2vm\" (UniqueName: \"kubernetes.io/projected/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-kube-api-access-rw2vm\") pod \"ipp-route-8291ef-kserve-586b75864c-k4dhq\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") " pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.522178 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.522155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:33.799383 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:33.799252    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"]
Sep 10 21:45:33.801980 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:45:33.801952    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd62a6e60_38ab_4a11_8242_a3a5980a0f8c.slice/crio-0721b3380ac8043d394e634fa8c115ea314debed3fc00d835224c00cb2afeab8 WatchSource:0}: Error finding container 0721b3380ac8043d394e634fa8c115ea314debed3fc00d835224c00cb2afeab8: Status 404 returned error can't find the container with id 0721b3380ac8043d394e634fa8c115ea314debed3fc00d835224c00cb2afeab8
Sep 10 21:45:34.565717 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:34.565681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq" event={"ID":"d62a6e60-38ab-4a11-8242-a3a5980a0f8c","Type":"ContainerStarted","Data":"f4c918e5cd1dd1ed88aac576e02180c711dd1a807b39fb521051380091ca4af0"}
Sep 10 21:45:34.565717 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:34.565722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq" event={"ID":"d62a6e60-38ab-4a11-8242-a3a5980a0f8c","Type":"ContainerStarted","Data":"0721b3380ac8043d394e634fa8c115ea314debed3fc00d835224c00cb2afeab8"}
Sep 10 21:45:34.584382 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:34.584328    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq" podStartSLOduration=1.584313904 podStartE2EDuration="1.584313904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:45:34.581577912 +0000 UTC m=+1573.229492090" watchObservedRunningTime="2026-09-10 21:45:34.584313904 +0000 UTC m=+1573.232228071"
Sep 10 21:45:37.308893 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:37.308848    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:45:43.523155 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:43.523121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:43.523628 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:43.523172    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:43.536281 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:43.536252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:45:43.623473 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:45:43.623443    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:46:05.008464 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:46:05.008427    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:46:11.228112 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:46:11.228082    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz_2dd648bb-3025-40be-b056-f171ff9ccd84/payload-processing/0.log"
Sep 10 21:46:11.485339 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:46:11.485248    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm_ef3673cd-0191-4f84-9f90-2f949c21f95c/payload-processing/0.log"
Sep 10 21:46:25.309183 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:46:25.309096    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:46:51.619238 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:46:51.619201    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:47:31.208986 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:31.208950    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:47:33.573975 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.573940    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"]
Sep 10 21:47:33.574391 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.574212    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz" podUID="2dd648bb-3025-40be-b056-f171ff9ccd84" containerName="payload-processing" containerID="cri-o://69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa" gracePeriod=30
Sep 10 21:47:33.582799 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.582770    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"]
Sep 10 21:47:33.583034 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.583007    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4" podUID="8e65351a-7acb-4339-8e46-7d32316f6c86" containerName="payload-pre-processing" containerID="cri-o://a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627" gracePeriod=30
Sep 10 21:47:33.833788 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.833730    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:47:33.856467 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.856443    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:47:33.935484 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.935451    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7bjc\" (UniqueName: \"kubernetes.io/projected/8e65351a-7acb-4339-8e46-7d32316f6c86-kube-api-access-c7bjc\") pod \"8e65351a-7acb-4339-8e46-7d32316f6c86\" (UID: \"8e65351a-7acb-4339-8e46-7d32316f6c86\") "
Sep 10 21:47:33.935677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.935516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e65351a-7acb-4339-8e46-7d32316f6c86-plugins-config-volume\") pod \"8e65351a-7acb-4339-8e46-7d32316f6c86\" (UID: \"8e65351a-7acb-4339-8e46-7d32316f6c86\") "
Sep 10 21:47:33.935677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.935563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd648bb-3025-40be-b056-f171ff9ccd84-plugins-config-volume\") pod \"2dd648bb-3025-40be-b056-f171ff9ccd84\" (UID: \"2dd648bb-3025-40be-b056-f171ff9ccd84\") "
Sep 10 21:47:33.935677 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.935582    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpfqx\" (UniqueName: \"kubernetes.io/projected/2dd648bb-3025-40be-b056-f171ff9ccd84-kube-api-access-wpfqx\") pod \"2dd648bb-3025-40be-b056-f171ff9ccd84\" (UID: \"2dd648bb-3025-40be-b056-f171ff9ccd84\") "
Sep 10 21:47:33.935937 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.935907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e65351a-7acb-4339-8e46-7d32316f6c86-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e65351a-7acb-4339-8e46-7d32316f6c86" (UID: "8e65351a-7acb-4339-8e46-7d32316f6c86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:47:33.936068 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.935997    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd648bb-3025-40be-b056-f171ff9ccd84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2dd648bb-3025-40be-b056-f171ff9ccd84" (UID: "2dd648bb-3025-40be-b056-f171ff9ccd84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:47:33.937857 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.937832    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e65351a-7acb-4339-8e46-7d32316f6c86-kube-api-access-c7bjc" (OuterVolumeSpecName: "kube-api-access-c7bjc") pod "8e65351a-7acb-4339-8e46-7d32316f6c86" (UID: "8e65351a-7acb-4339-8e46-7d32316f6c86"). InnerVolumeSpecName "kube-api-access-c7bjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:47:33.938262 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:33.938236    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd648bb-3025-40be-b056-f171ff9ccd84-kube-api-access-wpfqx" (OuterVolumeSpecName: "kube-api-access-wpfqx") pod "2dd648bb-3025-40be-b056-f171ff9ccd84" (UID: "2dd648bb-3025-40be-b056-f171ff9ccd84"). InnerVolumeSpecName "kube-api-access-wpfqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:47:34.036728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.036706    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7bjc\" (UniqueName: \"kubernetes.io/projected/8e65351a-7acb-4339-8e46-7d32316f6c86-kube-api-access-c7bjc\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:47:34.036728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.036727    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e65351a-7acb-4339-8e46-7d32316f6c86-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:47:34.036869 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.036736    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd648bb-3025-40be-b056-f171ff9ccd84-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:47:34.036869 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.036744    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpfqx\" (UniqueName: \"kubernetes.io/projected/2dd648bb-3025-40be-b056-f171ff9ccd84-kube-api-access-wpfqx\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:47:34.166839 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.166747    2580 generic.go:356] "Generic (PLEG): container finished" podID="8e65351a-7acb-4339-8e46-7d32316f6c86" containerID="a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627" exitCode=0
Sep 10 21:47:34.166839 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.166786    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"
Sep 10 21:47:34.167045 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.166833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4" event={"ID":"8e65351a-7acb-4339-8e46-7d32316f6c86","Type":"ContainerDied","Data":"a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627"}
Sep 10 21:47:34.167045 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.166886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4" event={"ID":"8e65351a-7acb-4339-8e46-7d32316f6c86","Type":"ContainerDied","Data":"fbb2f013810399f531c2b16d8be72aae40c83d3e53756514f95df3ddd61bced6"}
Sep 10 21:47:34.167045 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.166926    2580 scope.go:117] "RemoveContainer" containerID="a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627"
Sep 10 21:47:34.168187 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.168164    2580 generic.go:356] "Generic (PLEG): container finished" podID="2dd648bb-3025-40be-b056-f171ff9ccd84" containerID="69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa" exitCode=0
Sep 10 21:47:34.168290 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.168189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz" event={"ID":"2dd648bb-3025-40be-b056-f171ff9ccd84","Type":"ContainerDied","Data":"69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa"}
Sep 10 21:47:34.168290 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.168206    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"
Sep 10 21:47:34.168290 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.168229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz" event={"ID":"2dd648bb-3025-40be-b056-f171ff9ccd84","Type":"ContainerDied","Data":"6353bffae3f50b0410d612f37f1561ddd1922cfd5d771e35491da67ef02603cb"}
Sep 10 21:47:34.181355 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.178376    2580 scope.go:117] "RemoveContainer" containerID="a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627"
Sep 10 21:47:34.181447 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:47:34.181391    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627\": container with ID starting with a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627 not found: ID does not exist" containerID="a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627"
Sep 10 21:47:34.181509 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.181462    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627"} err="failed to get container status \"a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627\": rpc error: code = NotFound desc = could not find container \"a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627\": container with ID starting with a0d34936238cfe8d0343e8cdc3eadf0221965a9313a6ba4b9b0126559b097627 not found: ID does not exist"
Sep 10 21:47:34.181509 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.181485    2580 scope.go:117] "RemoveContainer" containerID="69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa"
Sep 10 21:47:34.189889 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.189871    2580 scope.go:117] "RemoveContainer" containerID="69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa"
Sep 10 21:47:34.190161 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:47:34.190143    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa\": container with ID starting with 69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa not found: ID does not exist" containerID="69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa"
Sep 10 21:47:34.190215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.190158    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"]
Sep 10 21:47:34.190215 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.190167    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa"} err="failed to get container status \"69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa\": rpc error: code = NotFound desc = could not find container \"69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa\": container with ID starting with 69debb962bbab72eea6768924a748ca310351c19089ca7ca738f4346ca55a2fa not found: ID does not exist"
Sep 10 21:47:34.195635 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.195616    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8291ef-685747d666-wrxt4"]
Sep 10 21:47:34.206373 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.206353    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"]
Sep 10 21:47:34.212425 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:34.212393    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8291ef-c9f94fd4c-9lhqz"]
Sep 10 21:47:35.820354 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:35.820313    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd648bb-3025-40be-b056-f171ff9ccd84" path="/var/lib/kubelet/pods/2dd648bb-3025-40be-b056-f171ff9ccd84/volumes"
Sep 10 21:47:35.820752 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:35.820723    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e65351a-7acb-4339-8e46-7d32316f6c86" path="/var/lib/kubelet/pods/8e65351a-7acb-4339-8e46-7d32316f6c86/volumes"
Sep 10 21:47:39.096216 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:39.096182    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"]
Sep 10 21:47:44.316383 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:44.316348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:47:54.115671 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.115612    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq" podUID="d62a6e60-38ab-4a11-8242-a3a5980a0f8c" containerName="main" containerID="cri-o://f4c918e5cd1dd1ed88aac576e02180c711dd1a807b39fb521051380091ca4af0" gracePeriod=45
Sep 10 21:47:54.274558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.274519    2580 generic.go:356] "Generic (PLEG): container finished" podID="d62a6e60-38ab-4a11-8242-a3a5980a0f8c" containerID="f4c918e5cd1dd1ed88aac576e02180c711dd1a807b39fb521051380091ca4af0" exitCode=0
Sep 10 21:47:54.274716 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.274592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq" event={"ID":"d62a6e60-38ab-4a11-8242-a3a5980a0f8c","Type":"ContainerDied","Data":"f4c918e5cd1dd1ed88aac576e02180c711dd1a807b39fb521051380091ca4af0"}
Sep 10 21:47:54.363424 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.363377    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:47:54.536109 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.536075    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw2vm\" (UniqueName: \"kubernetes.io/projected/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-kube-api-access-rw2vm\") pod \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") "
Sep 10 21:47:54.536298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.536143    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-dshm\") pod \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") "
Sep 10 21:47:54.536298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.536170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tmp-dir\") pod \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") "
Sep 10 21:47:54.536298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.536203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-model-cache\") pod \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") "
Sep 10 21:47:54.536298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.536222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-home\") pod \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") "
Sep 10 21:47:54.536298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.536301    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tls-certs\") pod \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\" (UID: \"d62a6e60-38ab-4a11-8242-a3a5980a0f8c\") "
Sep 10 21:47:54.536627 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.536494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d62a6e60-38ab-4a11-8242-a3a5980a0f8c" (UID: "d62a6e60-38ab-4a11-8242-a3a5980a0f8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:47:54.536627 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.536509    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d62a6e60-38ab-4a11-8242-a3a5980a0f8c" (UID: "d62a6e60-38ab-4a11-8242-a3a5980a0f8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:47:54.536627 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.536550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-home" (OuterVolumeSpecName: "home") pod "d62a6e60-38ab-4a11-8242-a3a5980a0f8c" (UID: "d62a6e60-38ab-4a11-8242-a3a5980a0f8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:47:54.538616 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.538589    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d62a6e60-38ab-4a11-8242-a3a5980a0f8c" (UID: "d62a6e60-38ab-4a11-8242-a3a5980a0f8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:47:54.538713 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.538620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-dshm" (OuterVolumeSpecName: "dshm") pod "d62a6e60-38ab-4a11-8242-a3a5980a0f8c" (UID: "d62a6e60-38ab-4a11-8242-a3a5980a0f8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:47:54.538767 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.538727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-kube-api-access-rw2vm" (OuterVolumeSpecName: "kube-api-access-rw2vm") pod "d62a6e60-38ab-4a11-8242-a3a5980a0f8c" (UID: "d62a6e60-38ab-4a11-8242-a3a5980a0f8c"). InnerVolumeSpecName "kube-api-access-rw2vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:47:54.637119 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.637088    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:47:54.637119 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.637114    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:47:54.637119 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.637123    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:47:54.637341 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.637132    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw2vm\" (UniqueName: \"kubernetes.io/projected/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-kube-api-access-rw2vm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:47:54.637341 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.637141    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:47:54.637341 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:54.637153    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62a6e60-38ab-4a11-8242-a3a5980a0f8c-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:47:55.281740 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:55.281707    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"
Sep 10 21:47:55.282226 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:55.281743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq" event={"ID":"d62a6e60-38ab-4a11-8242-a3a5980a0f8c","Type":"ContainerDied","Data":"0721b3380ac8043d394e634fa8c115ea314debed3fc00d835224c00cb2afeab8"}
Sep 10 21:47:55.282226 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:55.281795    2580 scope.go:117] "RemoveContainer" containerID="f4c918e5cd1dd1ed88aac576e02180c711dd1a807b39fb521051380091ca4af0"
Sep 10 21:47:55.302895 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:55.302857    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"]
Sep 10 21:47:55.306652 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:55.306603    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8291ef/ipp-route-8291ef-kserve-586b75864c-k4dhq"]
Sep 10 21:47:55.822562 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:47:55.822531    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d62a6e60-38ab-4a11-8242-a3a5980a0f8c" path="/var/lib/kubelet/pods/d62a6e60-38ab-4a11-8242-a3a5980a0f8c/volumes"
Sep 10 21:48:33.122244 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.122208    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"]
Sep 10 21:48:33.122821 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.122528    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm" podUID="ef3673cd-0191-4f84-9f90-2f949c21f95c" containerName="payload-processing" containerID="cri-o://4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a" gracePeriod=30
Sep 10 21:48:33.377268 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.377195    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:48:33.483547 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.483510    2580 generic.go:356] "Generic (PLEG): container finished" podID="ef3673cd-0191-4f84-9f90-2f949c21f95c" containerID="4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a" exitCode=0
Sep 10 21:48:33.483728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.483584    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"
Sep 10 21:48:33.483728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.483592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm" event={"ID":"ef3673cd-0191-4f84-9f90-2f949c21f95c","Type":"ContainerDied","Data":"4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a"}
Sep 10 21:48:33.483863 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.483731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm" event={"ID":"ef3673cd-0191-4f84-9f90-2f949c21f95c","Type":"ContainerDied","Data":"1bef9983cf20ac7a3e6e37fcb2abad4a420c8720ee6ab34e8169f698c0732e61"}
Sep 10 21:48:33.483863 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.483759    2580 scope.go:117] "RemoveContainer" containerID="4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a"
Sep 10 21:48:33.492922 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.492902    2580 scope.go:117] "RemoveContainer" containerID="4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a"
Sep 10 21:48:33.493218 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:48:33.493197    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a\": container with ID starting with 4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a not found: ID does not exist" containerID="4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a"
Sep 10 21:48:33.493289 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.493231    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a"} err="failed to get container status \"4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a\": rpc error: code = NotFound desc = could not find container \"4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a\": container with ID starting with 4ab1898fa5f3ecfa08d6cd9e9bf05dad8890c2537d9b7830c39651598ca6ee4a not found: ID does not exist"
Sep 10 21:48:33.504634 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.504610    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tvxc\" (UniqueName: \"kubernetes.io/projected/ef3673cd-0191-4f84-9f90-2f949c21f95c-kube-api-access-5tvxc\") pod \"ef3673cd-0191-4f84-9f90-2f949c21f95c\" (UID: \"ef3673cd-0191-4f84-9f90-2f949c21f95c\") "
Sep 10 21:48:33.504703 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.504651    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3673cd-0191-4f84-9f90-2f949c21f95c-plugins-config-volume\") pod \"ef3673cd-0191-4f84-9f90-2f949c21f95c\" (UID: \"ef3673cd-0191-4f84-9f90-2f949c21f95c\") "
Sep 10 21:48:33.505017 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.504997    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3673cd-0191-4f84-9f90-2f949c21f95c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef3673cd-0191-4f84-9f90-2f949c21f95c" (UID: "ef3673cd-0191-4f84-9f90-2f949c21f95c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:48:33.506842 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.506822    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3673cd-0191-4f84-9f90-2f949c21f95c-kube-api-access-5tvxc" (OuterVolumeSpecName: "kube-api-access-5tvxc") pod "ef3673cd-0191-4f84-9f90-2f949c21f95c" (UID: "ef3673cd-0191-4f84-9f90-2f949c21f95c"). InnerVolumeSpecName "kube-api-access-5tvxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:48:33.606192 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.606148    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tvxc\" (UniqueName: \"kubernetes.io/projected/ef3673cd-0191-4f84-9f90-2f949c21f95c-kube-api-access-5tvxc\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:33.606192 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.606184    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3673cd-0191-4f84-9f90-2f949c21f95c-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:33.805766 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.805728    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"]
Sep 10 21:48:33.809745 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.809711    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2fbf53-7bb99d9f67-lmjfm"]
Sep 10 21:48:33.821998 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:33.821969    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef3673cd-0191-4f84-9f90-2f949c21f95c" path="/var/lib/kubelet/pods/ef3673cd-0191-4f84-9f90-2f949c21f95c/volumes"
Sep 10 21:48:36.417185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:36.417143    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:48:39.885142 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:39.885099    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"]
Sep 10 21:48:39.885634 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:39.885321    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy" containerID="cri-o://821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253" gracePeriod=30
Sep 10 21:48:45.139372 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.139335    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:48:45.221758 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.221718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrsqz\" (UniqueName: \"kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-kube-api-access-wrsqz\") pod \"23ade033-3cca-4b16-9cc1-558e416bd53b\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") "
Sep 10 21:48:45.221967 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.221766    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-data\") pod \"23ade033-3cca-4b16-9cc1-558e416bd53b\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") "
Sep 10 21:48:45.221967 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.221826    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ade033-3cca-4b16-9cc1-558e416bd53b-istiod-ca-cert\") pod \"23ade033-3cca-4b16-9cc1-558e416bd53b\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") "
Sep 10 21:48:45.221967 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.221851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-socket\") pod \"23ade033-3cca-4b16-9cc1-558e416bd53b\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") "
Sep 10 21:48:45.221967 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.221895    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-envoy\") pod \"23ade033-3cca-4b16-9cc1-558e416bd53b\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") "
Sep 10 21:48:45.221967 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.221917    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-podinfo\") pod \"23ade033-3cca-4b16-9cc1-558e416bd53b\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") "
Sep 10 21:48:45.221967 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.221936    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-certs\") pod \"23ade033-3cca-4b16-9cc1-558e416bd53b\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") "
Sep 10 21:48:45.222285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.221983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-token\") pod \"23ade033-3cca-4b16-9cc1-558e416bd53b\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") "
Sep 10 21:48:45.222285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.222032    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-credential-socket\") pod \"23ade033-3cca-4b16-9cc1-558e416bd53b\" (UID: \"23ade033-3cca-4b16-9cc1-558e416bd53b\") "
Sep 10 21:48:45.222285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.222132    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-data" (OuterVolumeSpecName: "istio-data") pod "23ade033-3cca-4b16-9cc1-558e416bd53b" (UID: "23ade033-3cca-4b16-9cc1-558e416bd53b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:48:45.222285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.222182    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "23ade033-3cca-4b16-9cc1-558e416bd53b" (UID: "23ade033-3cca-4b16-9cc1-558e416bd53b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:48:45.222285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.222233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ade033-3cca-4b16-9cc1-558e416bd53b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "23ade033-3cca-4b16-9cc1-558e416bd53b" (UID: "23ade033-3cca-4b16-9cc1-558e416bd53b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:48:45.222936 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.222348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "23ade033-3cca-4b16-9cc1-558e416bd53b" (UID: "23ade033-3cca-4b16-9cc1-558e416bd53b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:48:45.222936 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.222360    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "23ade033-3cca-4b16-9cc1-558e416bd53b" (UID: "23ade033-3cca-4b16-9cc1-558e416bd53b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:48:45.222936 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.222503    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ade033-3cca-4b16-9cc1-558e416bd53b-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:45.222936 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.222533    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:45.222936 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.222547    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:45.222936 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.222560    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:45.222936 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.222574    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:45.224538 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.224510    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "23ade033-3cca-4b16-9cc1-558e416bd53b" (UID: "23ade033-3cca-4b16-9cc1-558e416bd53b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:48:45.224683 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.224631    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "23ade033-3cca-4b16-9cc1-558e416bd53b" (UID: "23ade033-3cca-4b16-9cc1-558e416bd53b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:48:45.224868 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.224845    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-kube-api-access-wrsqz" (OuterVolumeSpecName: "kube-api-access-wrsqz") pod "23ade033-3cca-4b16-9cc1-558e416bd53b" (UID: "23ade033-3cca-4b16-9cc1-558e416bd53b"). InnerVolumeSpecName "kube-api-access-wrsqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:48:45.224936 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.224922    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-token" (OuterVolumeSpecName: "istio-token") pod "23ade033-3cca-4b16-9cc1-558e416bd53b" (UID: "23ade033-3cca-4b16-9cc1-558e416bd53b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:48:45.323857 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.323812    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrsqz\" (UniqueName: \"kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-kube-api-access-wrsqz\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:45.323857 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.323851    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:45.323857 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.323861    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:45.323857 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.323869    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ade033-3cca-4b16-9cc1-558e416bd53b-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:45.547261 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.547166    2580 generic.go:356] "Generic (PLEG): container finished" podID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerID="821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253" exitCode=0
Sep 10 21:48:45.547261 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.547223    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"
Sep 10 21:48:45.547499 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.547257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" event={"ID":"23ade033-3cca-4b16-9cc1-558e416bd53b","Type":"ContainerDied","Data":"821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253"}
Sep 10 21:48:45.547499 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.547354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr" event={"ID":"23ade033-3cca-4b16-9cc1-558e416bd53b","Type":"ContainerDied","Data":"3c19dc2ddcd578aed8c7f7ce08aa5dc6c21a82ae5b141d36aa894b991e3af4f6"}
Sep 10 21:48:45.547499 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.547384    2580 scope.go:117] "RemoveContainer" containerID="821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253"
Sep 10 21:48:45.557224 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.557204    2580 scope.go:117] "RemoveContainer" containerID="7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2"
Sep 10 21:48:45.567236 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.567206    2580 scope.go:117] "RemoveContainer" containerID="821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253"
Sep 10 21:48:45.567641 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:48:45.567601    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253\": container with ID starting with 821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253 not found: ID does not exist" containerID="821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253"
Sep 10 21:48:45.567696 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.567654    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253"} err="failed to get container status \"821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253\": rpc error: code = NotFound desc = could not find container \"821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253\": container with ID starting with 821ac3f80b434bb75291b959509b874f0f34079602b26cdbdabfcf29d5aab253 not found: ID does not exist"
Sep 10 21:48:45.567696 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.567675    2580 scope.go:117] "RemoveContainer" containerID="7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2"
Sep 10 21:48:45.567997 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:48:45.567979    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2\": container with ID starting with 7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2 not found: ID does not exist" containerID="7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2"
Sep 10 21:48:45.568069 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.567998    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2"} err="failed to get container status \"7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2\": rpc error: code = NotFound desc = could not find container \"7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2\": container with ID starting with 7285ed1d0bbd7b352fff473c11b39769440c772b002b52a2779aad96455a0ec2 not found: ID does not exist"
Sep 10 21:48:45.581580 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.581540    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"]
Sep 10 21:48:45.599835 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.599791    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2fbf53-openshift-default-57585b5688-lk7sr"]
Sep 10 21:48:45.820887 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:45.820801    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" path="/var/lib/kubelet/pods/23ade033-3cca-4b16-9cc1-558e416bd53b/volumes"
Sep 10 21:48:50.682286 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:50.682250    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"]
Sep 10 21:48:50.682782 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:50.682514    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f" podUID="25316051-d336-49b0-b1c5-e997d8ec5d8b" containerName="maas-api" containerID="cri-o://8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19" gracePeriod=30
Sep 10 21:48:50.722190 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:50.722151    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4"]
Sep 10 21:48:50.725567 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:50.725526    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2d9e68-29817945-rqsv4"]
Sep 10 21:48:50.752780 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:50.752732    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f" podUID="25316051-d336-49b0-b1c5-e997d8ec5d8b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.50:8443/health\": dial tcp 10.133.0.50:8443: connect: connection refused"
Sep 10 21:48:50.754656 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:50.754493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"]
Sep 10 21:48:50.754889 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:50.754812    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b" podUID="26e0dd97-a253-48c6-965d-70921a610bce" containerName="payload-pre-processing" containerID="cri-o://c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7" gracePeriod=30
Sep 10 21:48:51.009481 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.009444    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:48:51.037661 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.037631    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:48:51.079096 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.079065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7slxw\" (UniqueName: \"kubernetes.io/projected/26e0dd97-a253-48c6-965d-70921a610bce-kube-api-access-7slxw\") pod \"26e0dd97-a253-48c6-965d-70921a610bce\" (UID: \"26e0dd97-a253-48c6-965d-70921a610bce\") "
Sep 10 21:48:51.079274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.079111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-metrics-certs\") pod \"25316051-d336-49b0-b1c5-e997d8ec5d8b\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") "
Sep 10 21:48:51.079274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.079143    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-maas-api-tls\") pod \"25316051-d336-49b0-b1c5-e997d8ec5d8b\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") "
Sep 10 21:48:51.079274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.079215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25316051-d336-49b0-b1c5-e997d8ec5d8b-openshift-service-ca\") pod \"25316051-d336-49b0-b1c5-e997d8ec5d8b\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") "
Sep 10 21:48:51.079439 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.079284    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e0dd97-a253-48c6-965d-70921a610bce-plugins-config-volume\") pod \"26e0dd97-a253-48c6-965d-70921a610bce\" (UID: \"26e0dd97-a253-48c6-965d-70921a610bce\") "
Sep 10 21:48:51.079439 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.079323    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8rzc\" (UniqueName: \"kubernetes.io/projected/25316051-d336-49b0-b1c5-e997d8ec5d8b-kube-api-access-x8rzc\") pod \"25316051-d336-49b0-b1c5-e997d8ec5d8b\" (UID: \"25316051-d336-49b0-b1c5-e997d8ec5d8b\") "
Sep 10 21:48:51.079725 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.079691    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25316051-d336-49b0-b1c5-e997d8ec5d8b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25316051-d336-49b0-b1c5-e997d8ec5d8b" (UID: "25316051-d336-49b0-b1c5-e997d8ec5d8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:48:51.079880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.079809    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e0dd97-a253-48c6-965d-70921a610bce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26e0dd97-a253-48c6-965d-70921a610bce" (UID: "26e0dd97-a253-48c6-965d-70921a610bce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:48:51.081788 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.081757    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25316051-d336-49b0-b1c5-e997d8ec5d8b" (UID: "25316051-d336-49b0-b1c5-e997d8ec5d8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:48:51.082060 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.082030    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e0dd97-a253-48c6-965d-70921a610bce-kube-api-access-7slxw" (OuterVolumeSpecName: "kube-api-access-7slxw") pod "26e0dd97-a253-48c6-965d-70921a610bce" (UID: "26e0dd97-a253-48c6-965d-70921a610bce"). InnerVolumeSpecName "kube-api-access-7slxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:48:51.082152 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.082100    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "25316051-d336-49b0-b1c5-e997d8ec5d8b" (UID: "25316051-d336-49b0-b1c5-e997d8ec5d8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:48:51.082203 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.082188    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25316051-d336-49b0-b1c5-e997d8ec5d8b-kube-api-access-x8rzc" (OuterVolumeSpecName: "kube-api-access-x8rzc") pod "25316051-d336-49b0-b1c5-e997d8ec5d8b" (UID: "25316051-d336-49b0-b1c5-e997d8ec5d8b"). InnerVolumeSpecName "kube-api-access-x8rzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:48:51.180997 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.180955    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-maas-api-tls\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:51.180997 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.180989    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25316051-d336-49b0-b1c5-e997d8ec5d8b-openshift-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:51.180997 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.181000    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e0dd97-a253-48c6-965d-70921a610bce-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:51.180997 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.181009    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8rzc\" (UniqueName: \"kubernetes.io/projected/25316051-d336-49b0-b1c5-e997d8ec5d8b-kube-api-access-x8rzc\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:51.181288 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.181017    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7slxw\" (UniqueName: \"kubernetes.io/projected/26e0dd97-a253-48c6-965d-70921a610bce-kube-api-access-7slxw\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:51.181288 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.181033    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25316051-d336-49b0-b1c5-e997d8ec5d8b-metrics-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:48:51.582990 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.582947    2580 generic.go:356] "Generic (PLEG): container finished" podID="26e0dd97-a253-48c6-965d-70921a610bce" containerID="c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7" exitCode=0
Sep 10 21:48:51.583204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.583006    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"
Sep 10 21:48:51.583204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.583038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b" event={"ID":"26e0dd97-a253-48c6-965d-70921a610bce","Type":"ContainerDied","Data":"c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7"}
Sep 10 21:48:51.583204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.583096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b" event={"ID":"26e0dd97-a253-48c6-965d-70921a610bce","Type":"ContainerDied","Data":"c51abada3db7e3a7d172d0eb9d4fdac6e3038ca98c2d786e118538d2c4b4e246"}
Sep 10 21:48:51.583204 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.583114    2580 scope.go:117] "RemoveContainer" containerID="c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7"
Sep 10 21:48:51.584648 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.584622    2580 generic.go:356] "Generic (PLEG): container finished" podID="25316051-d336-49b0-b1c5-e997d8ec5d8b" containerID="8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19" exitCode=0
Sep 10 21:48:51.584765 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.584680    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"
Sep 10 21:48:51.584765 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.584705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f" event={"ID":"25316051-d336-49b0-b1c5-e997d8ec5d8b","Type":"ContainerDied","Data":"8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19"}
Sep 10 21:48:51.584765 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.584758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f" event={"ID":"25316051-d336-49b0-b1c5-e997d8ec5d8b","Type":"ContainerDied","Data":"07b6916e2d5e32a533510aa97bccc4f6b7bafac0284c23adee61033e84bd6d26"}
Sep 10 21:48:51.596711 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.596481    2580 scope.go:117] "RemoveContainer" containerID="c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7"
Sep 10 21:48:51.597456 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:48:51.597403    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7\": container with ID starting with c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7 not found: ID does not exist" containerID="c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7"
Sep 10 21:48:51.597637 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.597481    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7"} err="failed to get container status \"c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7\": rpc error: code = NotFound desc = could not find container \"c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7\": container with ID starting with c1e0711fe0cda1976dedf50bbdcc3a8d9e2800e9124039c221c54f7b00ea59d7 not found: ID does not exist"
Sep 10 21:48:51.597637 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.597500    2580 scope.go:117] "RemoveContainer" containerID="8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19"
Sep 10 21:48:51.607130 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.607089    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"]
Sep 10 21:48:51.607483 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.607464    2580 scope.go:117] "RemoveContainer" containerID="8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19"
Sep 10 21:48:51.607841 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:48:51.607820    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19\": container with ID starting with 8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19 not found: ID does not exist" containerID="8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19"
Sep 10 21:48:51.607913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.607854    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19"} err="failed to get container status \"8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19\": rpc error: code = NotFound desc = could not find container \"8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19\": container with ID starting with 8a7cac5436cced10702ee67b960ef47fa1c5949dcaaeb17c3a67a6f55866de19 not found: ID does not exist"
Sep 10 21:48:51.611608 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.611569    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2d9e68-5c9745ff7d-vlg7b"]
Sep 10 21:48:51.622390 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.622354    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"]
Sep 10 21:48:51.626927 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.626895    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2d9e68-7f6f756856-bk85f"]
Sep 10 21:48:51.821498 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.821462    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25316051-d336-49b0-b1c5-e997d8ec5d8b" path="/var/lib/kubelet/pods/25316051-d336-49b0-b1c5-e997d8ec5d8b/volumes"
Sep 10 21:48:51.821897 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.821835    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e0dd97-a253-48c6-965d-70921a610bce" path="/var/lib/kubelet/pods/26e0dd97-a253-48c6-965d-70921a610bce/volumes"
Sep 10 21:48:51.822082 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:51.822068    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c471c2ba-5c4e-45a5-991a-7c458bfc818d" path="/var/lib/kubelet/pods/c471c2ba-5c4e-45a5-991a-7c458bfc818d/volumes"
Sep 10 21:48:53.610294 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:53.610256    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:48:57.721949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:57.721908    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"]
Sep 10 21:48:57.722383 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:57.722205    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy" containerID="cri-o://ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0" gracePeriod=30
Sep 10 21:48:58.074319 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074214    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"]
Sep 10 21:48:58.074612 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074598    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e0dd97-a253-48c6-965d-70921a610bce" containerName="payload-pre-processing"
Sep 10 21:48:58.074669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074614    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e0dd97-a253-48c6-965d-70921a610bce" containerName="payload-pre-processing"
Sep 10 21:48:58.074669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074626    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy"
Sep 10 21:48:58.074669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074632    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy"
Sep 10 21:48:58.074669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074641    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e65351a-7acb-4339-8e46-7d32316f6c86" containerName="payload-pre-processing"
Sep 10 21:48:58.074669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074647    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e65351a-7acb-4339-8e46-7d32316f6c86" containerName="payload-pre-processing"
Sep 10 21:48:58.074669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074653    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62a6e60-38ab-4a11-8242-a3a5980a0f8c" containerName="main"
Sep 10 21:48:58.074669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074658    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62a6e60-38ab-4a11-8242-a3a5980a0f8c" containerName="main"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074674    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3673cd-0191-4f84-9f90-2f949c21f95c" containerName="payload-processing"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074679    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3673cd-0191-4f84-9f90-2f949c21f95c" containerName="payload-processing"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074686    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25316051-d336-49b0-b1c5-e997d8ec5d8b" containerName="maas-api"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074691    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="25316051-d336-49b0-b1c5-e997d8ec5d8b" containerName="maas-api"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074697    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd648bb-3025-40be-b056-f171ff9ccd84" containerName="payload-processing"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074702    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd648bb-3025-40be-b056-f171ff9ccd84" containerName="payload-processing"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074711    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074717    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074773    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074781    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e0dd97-a253-48c6-965d-70921a610bce" containerName="payload-pre-processing"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074786    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="25316051-d336-49b0-b1c5-e997d8ec5d8b" containerName="maas-api"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074792    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e65351a-7acb-4339-8e46-7d32316f6c86" containerName="payload-pre-processing"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074797    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd648bb-3025-40be-b056-f171ff9ccd84" containerName="payload-processing"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074802    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62a6e60-38ab-4a11-8242-a3a5980a0f8c" containerName="main"
Sep 10 21:48:58.074913 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.074808    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3673cd-0191-4f84-9f90-2f949c21f95c" containerName="payload-processing"
Sep 10 21:48:58.078000 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.077977    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"
Sep 10 21:48:58.080240 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.080207    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wg7l\""
Sep 10 21:48:58.080404 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.080209    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:48:58.084685 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.084649    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"]
Sep 10 21:48:58.148725 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.148678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817052c-20f5-4786-8fc0-14002f94de5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6\" (UID: \"9817052c-20f5-4786-8fc0-14002f94de5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"
Sep 10 21:48:58.249965 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.249922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817052c-20f5-4786-8fc0-14002f94de5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6\" (UID: \"9817052c-20f5-4786-8fc0-14002f94de5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"
Sep 10 21:48:58.250558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.250535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817052c-20f5-4786-8fc0-14002f94de5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6\" (UID: \"9817052c-20f5-4786-8fc0-14002f94de5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"
Sep 10 21:48:58.392041 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.391941    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"
Sep 10 21:48:58.668780 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.668671    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"]
Sep 10 21:48:58.671095 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:48:58.671064    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9817052c_20f5_4786_8fc0_14002f94de5b.slice/crio-c58d1028b2fde8e0d0f2d250ead9a172f39b008dc0bb640574475d35aa2b8226 WatchSource:0}: Error finding container c58d1028b2fde8e0d0f2d250ead9a172f39b008dc0bb640574475d35aa2b8226: Status 404 returned error can't find the container with id c58d1028b2fde8e0d0f2d250ead9a172f39b008dc0bb640574475d35aa2b8226
Sep 10 21:48:58.899791 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.899756    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:48:58.900193 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:58.899815    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:48:59.631575 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:59.631541    2580 generic.go:356] "Generic (PLEG): container finished" podID="9817052c-20f5-4786-8fc0-14002f94de5b" containerID="1d0938711e88bc5a6e7cdf032b15783b3821ea1b95c320a06fe39123ccf825e2" exitCode=0
Sep 10 21:48:59.631762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:59.631611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6" event={"ID":"9817052c-20f5-4786-8fc0-14002f94de5b","Type":"ContainerDied","Data":"1d0938711e88bc5a6e7cdf032b15783b3821ea1b95c320a06fe39123ccf825e2"}
Sep 10 21:48:59.631762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:48:59.631652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6" event={"ID":"9817052c-20f5-4786-8fc0-14002f94de5b","Type":"ContainerStarted","Data":"c58d1028b2fde8e0d0f2d250ead9a172f39b008dc0bb640574475d35aa2b8226"}
Sep 10 21:49:00.767357 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:00.767324    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"
Sep 10 21:49:00.873949 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:00.873909    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817052c-20f5-4786-8fc0-14002f94de5b-maas-api-service-ca\") pod \"9817052c-20f5-4786-8fc0-14002f94de5b\" (UID: \"9817052c-20f5-4786-8fc0-14002f94de5b\") "
Sep 10 21:49:00.874266 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:00.874243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9817052c-20f5-4786-8fc0-14002f94de5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9817052c-20f5-4786-8fc0-14002f94de5b" (UID: "9817052c-20f5-4786-8fc0-14002f94de5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:49:00.975052 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:00.975004    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817052c-20f5-4786-8fc0-14002f94de5b-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:49:01.643453 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:01.643390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6" event={"ID":"9817052c-20f5-4786-8fc0-14002f94de5b","Type":"ContainerDied","Data":"c58d1028b2fde8e0d0f2d250ead9a172f39b008dc0bb640574475d35aa2b8226"}
Sep 10 21:49:01.643453 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:01.643444    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"
Sep 10 21:49:01.643685 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:01.643466    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c58d1028b2fde8e0d0f2d250ead9a172f39b008dc0bb640574475d35aa2b8226"
Sep 10 21:49:02.985506 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:02.985469    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:49:03.096155 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096051    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-envoy\") pod \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") "
Sep 10 21:49:03.096155 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqpbq\" (UniqueName: \"kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-kube-api-access-kqpbq\") pod \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") "
Sep 10 21:49:03.096406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096166    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-podinfo\") pod \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") "
Sep 10 21:49:03.096406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-data\") pod \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") "
Sep 10 21:49:03.096406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096230    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-token\") pod \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") "
Sep 10 21:49:03.096406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096260    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-certs\") pod \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") "
Sep 10 21:49:03.096406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096277    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-credential-socket\") pod \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") "
Sep 10 21:49:03.096406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096298    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istiod-ca-cert\") pod \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") "
Sep 10 21:49:03.096406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096333    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-socket\") pod \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\" (UID: \"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b\") "
Sep 10 21:49:03.096805 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" (UID: "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:49:03.096805 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096647    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:49:03.096805 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" (UID: "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:49:03.096805 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" (UID: "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:49:03.096805 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096766    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" (UID: "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:49:03.097229 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.096925    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" (UID: "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:49:03.098686 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.098662    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" (UID: "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:49:03.098686 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.098674    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" (UID: "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:49:03.098852 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.098695    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" (UID: "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:49:03.098925 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.098911    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-kube-api-access-kqpbq" (OuterVolumeSpecName: "kube-api-access-kqpbq") pod "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" (UID: "5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b"). InnerVolumeSpecName "kube-api-access-kqpbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:49:03.197920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.197873    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:49:03.197920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.197913    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:49:03.197920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.197922    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:49:03.197920 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.197930    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:49:03.198196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.197940    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:49:03.198196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.197948    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:49:03.198196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.197956    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:49:03.198196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.197964    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqpbq\" (UniqueName: \"kubernetes.io/projected/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b-kube-api-access-kqpbq\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:49:03.655582 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.655549    2580 generic.go:356] "Generic (PLEG): container finished" podID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerID="ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0" exitCode=0
Sep 10 21:49:03.655769 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.655603    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"
Sep 10 21:49:03.655769 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.655631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" event={"ID":"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b","Type":"ContainerDied","Data":"ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0"}
Sep 10 21:49:03.655769 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.655691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq" event={"ID":"5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b","Type":"ContainerDied","Data":"7ef1f4f6f0bcd7e5e033e9d29cc097f7e0f19faed21fafb5499223babf2eabc2"}
Sep 10 21:49:03.655769 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.655723    2580 scope.go:117] "RemoveContainer" containerID="ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0"
Sep 10 21:49:03.665947 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.665922    2580 scope.go:117] "RemoveContainer" containerID="da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680"
Sep 10 21:49:03.674922 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.674898    2580 scope.go:117] "RemoveContainer" containerID="ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0"
Sep 10 21:49:03.675342 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:49:03.675319    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0\": container with ID starting with ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0 not found: ID does not exist" containerID="ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0"
Sep 10 21:49:03.675441 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.675353    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0"} err="failed to get container status \"ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0\": rpc error: code = NotFound desc = could not find container \"ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0\": container with ID starting with ac846f3419dd0fd3d1e19adfe9eefefb89a38134db5c435637ab39675dd1f2e0 not found: ID does not exist"
Sep 10 21:49:03.675441 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.675376    2580 scope.go:117] "RemoveContainer" containerID="da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680"
Sep 10 21:49:03.675725 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:49:03.675704    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680\": container with ID starting with da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680 not found: ID does not exist" containerID="da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680"
Sep 10 21:49:03.675778 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.675732    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680"} err="failed to get container status \"da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680\": rpc error: code = NotFound desc = could not find container \"da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680\": container with ID starting with da36aadcee7d589ccf2bf9ca511c63f15852226e4d49d24dc938e8acba160680 not found: ID does not exist"
Sep 10 21:49:03.694082 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.694037    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"]
Sep 10 21:49:03.700365 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.700325    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2d9e68-openshift-default-b5b9c9676-z24rq"]
Sep 10 21:49:03.820855 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:03.820817    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" path="/var/lib/kubelet/pods/5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b/volumes"
Sep 10 21:49:08.774497 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:08.774448    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"]
Sep 10 21:49:08.779270 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:08.779236    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-67d5e2-5rfq6"]
Sep 10 21:49:09.819754 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:09.819723    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9817052c-20f5-4786-8fc0-14002f94de5b" path="/var/lib/kubelet/pods/9817052c-20f5-4786-8fc0-14002f94de5b/volumes"
Sep 10 21:49:11.409330 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:11.409286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:49:29.312502 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:49:29.312464    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:50:51.116281 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:50:51.116236    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:51:14.822275 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:51:14.822229    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:51:22.092829 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:51:22.092787    2580 scope.go:117] "RemoveContainer" containerID="46c9df030dc47e8e89187f8501d7ff84c4e1a53bbca5b774b74a3850bc1954c8"
Sep 10 21:51:22.909436 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:51:22.909375    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:51:30.808621 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:51:30.808560    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:51:48.808504 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:51:48.808463    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:51:56.802595 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:51:56.802548    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:53:26.510001 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:26.509956    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:53:32.604618 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:32.604578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:53:41.806826 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.806781    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"]
Sep 10 21:53:41.807285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.807154    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy"
Sep 10 21:53:41.807285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.807170    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy"
Sep 10 21:53:41.807285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.807184    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9817052c-20f5-4786-8fc0-14002f94de5b" containerName="revoke-keys"
Sep 10 21:53:41.807285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.807192    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9817052c-20f5-4786-8fc0-14002f94de5b" containerName="revoke-keys"
Sep 10 21:53:41.807285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.807264    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy"
Sep 10 21:53:41.807285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.807276    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ade033-3cca-4b16-9cc1-558e416bd53b" containerName="istio-proxy"
Sep 10 21:53:41.807285 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.807289    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9817052c-20f5-4786-8fc0-14002f94de5b" containerName="revoke-keys"
Sep 10 21:53:41.810520 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.810496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:41.813103 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.813076    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-81abf60f-kserve-self-signed-certs\""
Sep 10 21:53:41.822546 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.822515    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"]
Sep 10 21:53:41.925738 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.925698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87rk\" (UniqueName: \"kubernetes.io/projected/0342b323-6fcc-421f-b0db-1909a0f8b60d-kube-api-access-w87rk\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:41.926053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.926022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-model-cache\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:41.926184 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.926086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0342b323-6fcc-421f-b0db-1909a0f8b60d-tls-certs\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:41.926184 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.926145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-home\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:41.926312 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.926187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-tmp-dir\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:41.926312 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:41.926230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-dshm\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.026788 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.026742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-model-cache\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.026965 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.026801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0342b323-6fcc-421f-b0db-1909a0f8b60d-tls-certs\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.026965 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.026921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-home\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.026965 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.026954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-tmp-dir\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.027096 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.026996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-dshm\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.027155 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.027111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w87rk\" (UniqueName: \"kubernetes.io/projected/0342b323-6fcc-421f-b0db-1909a0f8b60d-kube-api-access-w87rk\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.027155 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.027141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-model-cache\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.027270 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.027253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-home\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.027326 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.027308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-tmp-dir\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.029395 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.029372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-dshm\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.029565 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.029549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0342b323-6fcc-421f-b0db-1909a0f8b60d-tls-certs\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.039444 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.038661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87rk\" (UniqueName: \"kubernetes.io/projected/0342b323-6fcc-421f-b0db-1909a0f8b60d-kube-api-access-w87rk\") pod \"e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") " pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.121902 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.121805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:42.395792 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:42.395762    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"]
Sep 10 21:53:42.398130 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:53:42.398103    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0342b323_6fcc_421f_b0db_1909a0f8b60d.slice/crio-15f3f93a9d3432c72bc344aa1e39c5c68607ef3f53d5c7e8e8a13a75e79ca610 WatchSource:0}: Error finding container 15f3f93a9d3432c72bc344aa1e39c5c68607ef3f53d5c7e8e8a13a75e79ca610: Status 404 returned error can't find the container with id 15f3f93a9d3432c72bc344aa1e39c5c68607ef3f53d5c7e8e8a13a75e79ca610
Sep 10 21:53:43.011820 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:43.011724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5" event={"ID":"0342b323-6fcc-421f-b0db-1909a0f8b60d","Type":"ContainerStarted","Data":"4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517"}
Sep 10 21:53:43.011820 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:43.011772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5" event={"ID":"0342b323-6fcc-421f-b0db-1909a0f8b60d","Type":"ContainerStarted","Data":"15f3f93a9d3432c72bc344aa1e39c5c68607ef3f53d5c7e8e8a13a75e79ca610"}
Sep 10 21:53:43.028379 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:43.028313    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5" podStartSLOduration=2.028295317 podStartE2EDuration="2.028295317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:53:43.025925016 +0000 UTC m=+2061.673839185" watchObservedRunningTime="2026-09-10 21:53:43.028295317 +0000 UTC m=+2061.676209484"
Sep 10 21:53:44.407669 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:44.407624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:53:52.122195 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:52.122148    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:52.122738 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:52.122214    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:52.135073 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:52.135042    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:53:53.076167 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:53:53.076133    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:54:03.397273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.397233    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"]
Sep 10 21:54:03.397830 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.397807    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy"
Sep 10 21:54:03.397914 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.397834    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy"
Sep 10 21:54:03.397967 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.397934    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfb0d90-3326-41a7-b8c0-4b8d5bfb6d3b" containerName="istio-proxy"
Sep 10 21:54:03.400967 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.400944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.403355 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.403331    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-81abf60f-kserve-self-signed-certs\""
Sep 10 21:54:03.413377 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.413347    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"]
Sep 10 21:54:03.521104 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.521065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-model-cache\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.521318 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.521133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-home\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.521318 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.521183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-dshm\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.521318 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.521220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2gjr\" (UniqueName: \"kubernetes.io/projected/b876355d-e870-4166-9882-d8aae7097d91-kube-api-access-r2gjr\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.521318 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.521266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-tmp-dir\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.521524 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.521319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b876355d-e870-4166-9882-d8aae7097d91-tls-certs\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.622272 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.622222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-dshm\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.622532 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.622283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2gjr\" (UniqueName: \"kubernetes.io/projected/b876355d-e870-4166-9882-d8aae7097d91-kube-api-access-r2gjr\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.622532 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.622314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-tmp-dir\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.622532 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.622344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b876355d-e870-4166-9882-d8aae7097d91-tls-certs\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.622532 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.622401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-model-cache\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.622532 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.622486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-home\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.622829 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.622755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-tmp-dir\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.622829 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.622800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-home\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.622906 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.622839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-model-cache\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.624889 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.624868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-dshm\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.625048 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.625030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b876355d-e870-4166-9882-d8aae7097d91-tls-certs\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.631008 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.630988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2gjr\" (UniqueName: \"kubernetes.io/projected/b876355d-e870-4166-9882-d8aae7097d91-kube-api-access-r2gjr\") pod \"e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") " pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.711237 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.711198    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:03.987451 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:03.987397    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"]
Sep 10 21:54:03.990018 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:54:03.989983    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb876355d_e870_4166_9882_d8aae7097d91.slice/crio-facc4a36e0863c9c4d4c6017dc1ae89679e5ae67d1ed98ca943c3a1d2e591a46 WatchSource:0}: Error finding container facc4a36e0863c9c4d4c6017dc1ae89679e5ae67d1ed98ca943c3a1d2e591a46: Status 404 returned error can't find the container with id facc4a36e0863c9c4d4c6017dc1ae89679e5ae67d1ed98ca943c3a1d2e591a46
Sep 10 21:54:04.121534 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:04.121494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75" event={"ID":"b876355d-e870-4166-9882-d8aae7097d91","Type":"ContainerStarted","Data":"6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d"}
Sep 10 21:54:04.121728 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:04.121546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75" event={"ID":"b876355d-e870-4166-9882-d8aae7097d91","Type":"ContainerStarted","Data":"facc4a36e0863c9c4d4c6017dc1ae89679e5ae67d1ed98ca943c3a1d2e591a46"}
Sep 10 21:54:04.137132 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:04.137042    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75" podStartSLOduration=1.137022449 podStartE2EDuration="1.137022449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:54:04.13632815 +0000 UTC m=+2082.784242320" watchObservedRunningTime="2026-09-10 21:54:04.137022449 +0000 UTC m=+2082.784936617"
Sep 10 21:54:06.309860 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:06.309810    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:54:13.711513 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:13.711465    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:13.712027 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:13.711593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:13.724450 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:13.724391    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:14.185060 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:14.184960    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:21.163491 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:21.163355    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"]
Sep 10 21:54:22.853558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:22.853506    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"]
Sep 10 21:54:23.513719 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:23.513671    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:54:31.811433 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:31.811366    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:54:36.185123 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.185056    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5" podUID="0342b323-6fcc-421f-b0db-1909a0f8b60d" containerName="main" containerID="cri-o://4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517" gracePeriod=45
Sep 10 21:54:36.448900 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.448852    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:54:36.540239 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.540194    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-dshm\") pod \"0342b323-6fcc-421f-b0db-1909a0f8b60d\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") "
Sep 10 21:54:36.540461 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.540248    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0342b323-6fcc-421f-b0db-1909a0f8b60d-tls-certs\") pod \"0342b323-6fcc-421f-b0db-1909a0f8b60d\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") "
Sep 10 21:54:36.540461 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.540319    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-tmp-dir\") pod \"0342b323-6fcc-421f-b0db-1909a0f8b60d\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") "
Sep 10 21:54:36.540461 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.540400    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-model-cache\") pod \"0342b323-6fcc-421f-b0db-1909a0f8b60d\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") "
Sep 10 21:54:36.540598 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.540475    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w87rk\" (UniqueName: \"kubernetes.io/projected/0342b323-6fcc-421f-b0db-1909a0f8b60d-kube-api-access-w87rk\") pod \"0342b323-6fcc-421f-b0db-1909a0f8b60d\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") "
Sep 10 21:54:36.540598 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.540528    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-home\") pod \"0342b323-6fcc-421f-b0db-1909a0f8b60d\" (UID: \"0342b323-6fcc-421f-b0db-1909a0f8b60d\") "
Sep 10 21:54:36.540701 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.540676    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0342b323-6fcc-421f-b0db-1909a0f8b60d" (UID: "0342b323-6fcc-421f-b0db-1909a0f8b60d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:54:36.540701 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.540690    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0342b323-6fcc-421f-b0db-1909a0f8b60d" (UID: "0342b323-6fcc-421f-b0db-1909a0f8b60d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:54:36.540866 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.540844    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:36.540999 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.540869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-home" (OuterVolumeSpecName: "home") pod "0342b323-6fcc-421f-b0db-1909a0f8b60d" (UID: "0342b323-6fcc-421f-b0db-1909a0f8b60d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:54:36.540999 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.540883    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:36.542675 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.542649    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0342b323-6fcc-421f-b0db-1909a0f8b60d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0342b323-6fcc-421f-b0db-1909a0f8b60d" (UID: "0342b323-6fcc-421f-b0db-1909a0f8b60d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:54:36.543234 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.543197    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-dshm" (OuterVolumeSpecName: "dshm") pod "0342b323-6fcc-421f-b0db-1909a0f8b60d" (UID: "0342b323-6fcc-421f-b0db-1909a0f8b60d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:54:36.543234 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.543213    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0342b323-6fcc-421f-b0db-1909a0f8b60d-kube-api-access-w87rk" (OuterVolumeSpecName: "kube-api-access-w87rk") pod "0342b323-6fcc-421f-b0db-1909a0f8b60d" (UID: "0342b323-6fcc-421f-b0db-1909a0f8b60d"). InnerVolumeSpecName "kube-api-access-w87rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:54:36.641544 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.641498    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w87rk\" (UniqueName: \"kubernetes.io/projected/0342b323-6fcc-421f-b0db-1909a0f8b60d-kube-api-access-w87rk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:36.641544 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.641532    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:36.641544 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.641546    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0342b323-6fcc-421f-b0db-1909a0f8b60d-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:36.641544 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:36.641554    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0342b323-6fcc-421f-b0db-1909a0f8b60d-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:37.283274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:37.283230    2580 generic.go:356] "Generic (PLEG): container finished" podID="0342b323-6fcc-421f-b0db-1909a0f8b60d" containerID="4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517" exitCode=0
Sep 10 21:54:37.283274 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:37.283278    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"
Sep 10 21:54:37.283890 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:37.283320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5" event={"ID":"0342b323-6fcc-421f-b0db-1909a0f8b60d","Type":"ContainerDied","Data":"4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517"}
Sep 10 21:54:37.283890 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:37.283389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5" event={"ID":"0342b323-6fcc-421f-b0db-1909a0f8b60d","Type":"ContainerDied","Data":"15f3f93a9d3432c72bc344aa1e39c5c68607ef3f53d5c7e8e8a13a75e79ca610"}
Sep 10 21:54:37.283890 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:37.283434    2580 scope.go:117] "RemoveContainer" containerID="4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517"
Sep 10 21:54:37.293134 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:37.293113    2580 scope.go:117] "RemoveContainer" containerID="4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517"
Sep 10 21:54:37.293455 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:54:37.293435    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517\": container with ID starting with 4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517 not found: ID does not exist" containerID="4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517"
Sep 10 21:54:37.293504 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:37.293464    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517"} err="failed to get container status \"4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517\": rpc error: code = NotFound desc = could not find container \"4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517\": container with ID starting with 4aa2e1554a7c3003d47be63a1b4319aa31a3d54e255b7beb1f74cf2dc6a7b517 not found: ID does not exist"
Sep 10 21:54:37.305256 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:37.305218    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"]
Sep 10 21:54:37.309071 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:37.309030    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-81abf60f-kserve-5dc4c7df8c-4d5z5"]
Sep 10 21:54:37.820880 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:37.820834    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0342b323-6fcc-421f-b0db-1909a0f8b60d" path="/var/lib/kubelet/pods/0342b323-6fcc-421f-b0db-1909a0f8b60d/volumes"
Sep 10 21:54:37.874199 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:37.874161    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75" podUID="b876355d-e870-4166-9882-d8aae7097d91" containerName="main" containerID="cri-o://6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d" gracePeriod=45
Sep 10 21:54:38.123294 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.123250    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:38.155558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.155516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-dshm\") pod \"b876355d-e870-4166-9882-d8aae7097d91\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") "
Sep 10 21:54:38.155558 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.155563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-model-cache\") pod \"b876355d-e870-4166-9882-d8aae7097d91\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") "
Sep 10 21:54:38.155812 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.155625    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-tmp-dir\") pod \"b876355d-e870-4166-9882-d8aae7097d91\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") "
Sep 10 21:54:38.155812 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.155664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2gjr\" (UniqueName: \"kubernetes.io/projected/b876355d-e870-4166-9882-d8aae7097d91-kube-api-access-r2gjr\") pod \"b876355d-e870-4166-9882-d8aae7097d91\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") "
Sep 10 21:54:38.155812 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.155746    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b876355d-e870-4166-9882-d8aae7097d91-tls-certs\") pod \"b876355d-e870-4166-9882-d8aae7097d91\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") "
Sep 10 21:54:38.155812 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.155769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-home\") pod \"b876355d-e870-4166-9882-d8aae7097d91\" (UID: \"b876355d-e870-4166-9882-d8aae7097d91\") "
Sep 10 21:54:38.156022 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.155937    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-model-cache" (OuterVolumeSpecName: "model-cache") pod "b876355d-e870-4166-9882-d8aae7097d91" (UID: "b876355d-e870-4166-9882-d8aae7097d91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:54:38.156022 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.155952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b876355d-e870-4166-9882-d8aae7097d91" (UID: "b876355d-e870-4166-9882-d8aae7097d91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:54:38.156111 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.156071    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:38.156111 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.156091    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:38.156225 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.156163    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-home" (OuterVolumeSpecName: "home") pod "b876355d-e870-4166-9882-d8aae7097d91" (UID: "b876355d-e870-4166-9882-d8aae7097d91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:54:38.158053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.158022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b876355d-e870-4166-9882-d8aae7097d91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b876355d-e870-4166-9882-d8aae7097d91" (UID: "b876355d-e870-4166-9882-d8aae7097d91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:54:38.158053 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.158031    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-dshm" (OuterVolumeSpecName: "dshm") pod "b876355d-e870-4166-9882-d8aae7097d91" (UID: "b876355d-e870-4166-9882-d8aae7097d91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:54:38.158214 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.158070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b876355d-e870-4166-9882-d8aae7097d91-kube-api-access-r2gjr" (OuterVolumeSpecName: "kube-api-access-r2gjr") pod "b876355d-e870-4166-9882-d8aae7097d91" (UID: "b876355d-e870-4166-9882-d8aae7097d91"). InnerVolumeSpecName "kube-api-access-r2gjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:54:38.256578 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.256531    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b876355d-e870-4166-9882-d8aae7097d91-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:38.256578 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.256567    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:38.256578 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.256579    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b876355d-e870-4166-9882-d8aae7097d91-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:38.256833 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.256590    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2gjr\" (UniqueName: \"kubernetes.io/projected/b876355d-e870-4166-9882-d8aae7097d91-kube-api-access-r2gjr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:38.290648 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.290612    2580 generic.go:356] "Generic (PLEG): container finished" podID="b876355d-e870-4166-9882-d8aae7097d91" containerID="6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d" exitCode=0
Sep 10 21:54:38.291141 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.290660    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"
Sep 10 21:54:38.291141 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.290706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75" event={"ID":"b876355d-e870-4166-9882-d8aae7097d91","Type":"ContainerDied","Data":"6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d"}
Sep 10 21:54:38.291141 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.290747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75" event={"ID":"b876355d-e870-4166-9882-d8aae7097d91","Type":"ContainerDied","Data":"facc4a36e0863c9c4d4c6017dc1ae89679e5ae67d1ed98ca943c3a1d2e591a46"}
Sep 10 21:54:38.291141 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.290764    2580 scope.go:117] "RemoveContainer" containerID="6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d"
Sep 10 21:54:38.300622 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.300595    2580 scope.go:117] "RemoveContainer" containerID="6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d"
Sep 10 21:54:38.300988 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:54:38.300957    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d\": container with ID starting with 6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d not found: ID does not exist" containerID="6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d"
Sep 10 21:54:38.301066 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.300997    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d"} err="failed to get container status \"6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d\": rpc error: code = NotFound desc = could not find container \"6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d\": container with ID starting with 6842e034a5b5c68bc2856c329d85288c935753ec3cc618e0473af90fca973f9d not found: ID does not exist"
Sep 10 21:54:38.314955 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.314909    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"]
Sep 10 21:54:38.321455 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:38.321402    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-81abf60f-kserve-dbf664f47-pzg75"]
Sep 10 21:54:39.510662 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:39.510614    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:54:39.820928 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:39.820824    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b876355d-e870-4166-9882-d8aae7097d91" path="/var/lib/kubelet/pods/b876355d-e870-4166-9882-d8aae7097d91/volumes"
Sep 10 21:54:46.043643 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.043590    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:54:46.216703 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.216655    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-586c445589-qp7b4"]
Sep 10 21:54:46.217139 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.217118    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b876355d-e870-4166-9882-d8aae7097d91" containerName="main"
Sep 10 21:54:46.217229 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.217140    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b876355d-e870-4166-9882-d8aae7097d91" containerName="main"
Sep 10 21:54:46.217229 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.217164    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0342b323-6fcc-421f-b0db-1909a0f8b60d" containerName="main"
Sep 10 21:54:46.217229 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.217172    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0342b323-6fcc-421f-b0db-1909a0f8b60d" containerName="main"
Sep 10 21:54:46.217393 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.217267    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0342b323-6fcc-421f-b0db-1909a0f8b60d" containerName="main"
Sep 10 21:54:46.217393 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.217284    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b876355d-e870-4166-9882-d8aae7097d91" containerName="main"
Sep 10 21:54:46.221624 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.221593    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.226001 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.225972    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 21:54:46.226581 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.226557    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 21:54:46.226701 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.226561    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nmksg\""
Sep 10 21:54:46.226701 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.226663    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:54:46.230947 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.230918    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 21:54:46.232320 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.232288    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-586c445589-qp7b4"]
Sep 10 21:54:46.236116 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.236091    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"]
Sep 10 21:54:46.244824 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.244794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.256534 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.256503    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"]
Sep 10 21:54:46.327369 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.327253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146e35d0-7246-4987-afda-18eafbb93628-openshift-service-ca\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.327369 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.327301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-metrics-certs\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.327369 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.327335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-maas-api-tls\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.327719 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.327450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146e35d0-7246-4987-afda-18eafbb93628-metrics-certs\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.327719 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.327571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r2dz\" (UniqueName: \"kubernetes.io/projected/146e35d0-7246-4987-afda-18eafbb93628-kube-api-access-5r2dz\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.327719 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.327600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqzf\" (UniqueName: \"kubernetes.io/projected/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-kube-api-access-8zqzf\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.327719 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.327622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-openshift-service-ca\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.327719 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.327683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146e35d0-7246-4987-afda-18eafbb93628-maas-api-tls\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.428828 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.428787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r2dz\" (UniqueName: \"kubernetes.io/projected/146e35d0-7246-4987-afda-18eafbb93628-kube-api-access-5r2dz\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.428828 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.428826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqzf\" (UniqueName: \"kubernetes.io/projected/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-kube-api-access-8zqzf\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.429099 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.428846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-openshift-service-ca\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.429099 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.428867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146e35d0-7246-4987-afda-18eafbb93628-maas-api-tls\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.429099 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.428891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146e35d0-7246-4987-afda-18eafbb93628-openshift-service-ca\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.429099 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.428913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-metrics-certs\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.429099 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.428939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-maas-api-tls\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.429099 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.429001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146e35d0-7246-4987-afda-18eafbb93628-metrics-certs\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.429747 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.429686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146e35d0-7246-4987-afda-18eafbb93628-openshift-service-ca\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.429905 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.429874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-openshift-service-ca\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.431617 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.431591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146e35d0-7246-4987-afda-18eafbb93628-metrics-certs\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.431738 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.431684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146e35d0-7246-4987-afda-18eafbb93628-maas-api-tls\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.432185 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.432160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-maas-api-tls\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.432252 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.432169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-metrics-certs\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.437350 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.437321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqzf\" (UniqueName: \"kubernetes.io/projected/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-kube-api-access-8zqzf\") pod \"maas-api-586c445589-qp7b4\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") " pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.437518 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.437369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r2dz\" (UniqueName: \"kubernetes.io/projected/146e35d0-7246-4987-afda-18eafbb93628-kube-api-access-5r2dz\") pod \"maas-api-7f95b84469-r6knn\" (UID: \"146e35d0-7246-4987-afda-18eafbb93628\") " pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.533828 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.533786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:46.559353 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.559307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:46.821710 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.821681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-586c445589-qp7b4"]
Sep 10 21:54:46.824289 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:54:46.824262    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cf37a5_50e4_4455_a1d2_7dcd50aea1c3.slice/crio-d677a6d5cfd5f811e6ab3bf28a464c53b89543b96059861fd5eeb69a585ea6ce WatchSource:0}: Error finding container d677a6d5cfd5f811e6ab3bf28a464c53b89543b96059861fd5eeb69a585ea6ce: Status 404 returned error can't find the container with id d677a6d5cfd5f811e6ab3bf28a464c53b89543b96059861fd5eeb69a585ea6ce
Sep 10 21:54:46.825795 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.825770    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:54:46.848109 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:46.848011    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"]
Sep 10 21:54:46.851271 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:54:46.851239    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146e35d0_7246_4987_afda_18eafbb93628.slice/crio-bcd2a24fb99df702ea2b8e2fe272d24d53423d22fef292505d2c7d403e7d7135 WatchSource:0}: Error finding container bcd2a24fb99df702ea2b8e2fe272d24d53423d22fef292505d2c7d403e7d7135: Status 404 returned error can't find the container with id bcd2a24fb99df702ea2b8e2fe272d24d53423d22fef292505d2c7d403e7d7135
Sep 10 21:54:47.340156 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:47.340119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn" event={"ID":"146e35d0-7246-4987-afda-18eafbb93628","Type":"ContainerStarted","Data":"bcd2a24fb99df702ea2b8e2fe272d24d53423d22fef292505d2c7d403e7d7135"}
Sep 10 21:54:47.341774 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:47.341747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4" event={"ID":"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3","Type":"ContainerStarted","Data":"498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5"}
Sep 10 21:54:47.341922 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:47.341797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4" event={"ID":"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3","Type":"ContainerStarted","Data":"d677a6d5cfd5f811e6ab3bf28a464c53b89543b96059861fd5eeb69a585ea6ce"}
Sep 10 21:54:47.341922 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:47.341891    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:47.360800 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:47.360728    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4" podStartSLOduration=0.938386385 podStartE2EDuration="1.360706719s" totalImagesPullingTime="422.320334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:54:47.358764192 +0000 UTC m=+2126.006678361" watchObservedRunningTime="2026-09-10 21:54:47.360706719 +0000 UTC m=+2126.008620888"
Sep 10 21:54:48.349489 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:48.349443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn" event={"ID":"146e35d0-7246-4987-afda-18eafbb93628","Type":"ContainerStarted","Data":"57018286b7e26a10b73f980d6b49dc8273d75628b55abc6869e0258a52ff0769"}
Sep 10 21:54:48.349893 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:48.349762    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:48.370598 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:48.370535    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn" podStartSLOduration=1.858792711 podStartE2EDuration="2.370515043s" totalImagesPullingTime="511.722332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:54:48.368453767 +0000 UTC m=+2127.016367937" watchObservedRunningTime="2026-09-10 21:54:48.370515043 +0000 UTC m=+2127.018429211"
Sep 10 21:54:53.355879 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:53.355844    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:53.811798 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:53.811761    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:54:54.361318 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.361278    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f95b84469-r6knn"
Sep 10 21:54:54.417139 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.417103    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-586c445589-qp7b4"]
Sep 10 21:54:54.417405 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.417378    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4" podUID="e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3" containerName="maas-api" containerID="cri-o://498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5" gracePeriod=30
Sep 10 21:54:54.668592 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.668548    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:54.811375 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.811322    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-maas-api-tls\") pod \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") "
Sep 10 21:54:54.811375 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.811391    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-metrics-certs\") pod \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") "
Sep 10 21:54:54.811655 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.811470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-openshift-service-ca\") pod \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") "
Sep 10 21:54:54.811655 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.811547    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zqzf\" (UniqueName: \"kubernetes.io/projected/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-kube-api-access-8zqzf\") pod \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\" (UID: \"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3\") "
Sep 10 21:54:54.811883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.811855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3" (UID: "e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:54:54.813890 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.813855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3" (UID: "e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:54:54.814002 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.813887    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-kube-api-access-8zqzf" (OuterVolumeSpecName: "kube-api-access-8zqzf") pod "e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3" (UID: "e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3"). InnerVolumeSpecName "kube-api-access-8zqzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:54:54.814002 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.813867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3" (UID: "e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:54:54.912594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.912490    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-maas-api-tls\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:54.912594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.912527    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-metrics-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:54.912594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.912537    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-openshift-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:54.912594 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:54.912548    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zqzf\" (UniqueName: \"kubernetes.io/projected/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3-kube-api-access-8zqzf\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Sep 10 21:54:55.387889 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:55.387848    2580 generic.go:356] "Generic (PLEG): container finished" podID="e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3" containerID="498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5" exitCode=0
Sep 10 21:54:55.387889 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:55.387893    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4"
Sep 10 21:54:55.388389 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:55.387931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4" event={"ID":"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3","Type":"ContainerDied","Data":"498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5"}
Sep 10 21:54:55.388389 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:55.387983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-586c445589-qp7b4" event={"ID":"e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3","Type":"ContainerDied","Data":"d677a6d5cfd5f811e6ab3bf28a464c53b89543b96059861fd5eeb69a585ea6ce"}
Sep 10 21:54:55.388389 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:55.388005    2580 scope.go:117] "RemoveContainer" containerID="498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5"
Sep 10 21:54:55.398179 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:55.397979    2580 scope.go:117] "RemoveContainer" containerID="498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5"
Sep 10 21:54:55.398394 ip-10-0-135-226 kubenswrapper[2580]: E0910 21:54:55.398368    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5\": container with ID starting with 498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5 not found: ID does not exist" containerID="498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5"
Sep 10 21:54:55.398525 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:55.398405    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5"} err="failed to get container status \"498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5\": rpc error: code = NotFound desc = could not find container \"498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5\": container with ID starting with 498b6ce482585af13e7647fa8f4a6a040c980dcb69065c7cdacc96d4a32d75a5 not found: ID does not exist"
Sep 10 21:54:55.413933 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:55.413892    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-586c445589-qp7b4"]
Sep 10 21:54:55.416707 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:55.416674    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-586c445589-qp7b4"]
Sep 10 21:54:55.821104 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:54:55.821065    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3" path="/var/lib/kubelet/pods/e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3/volumes"
Sep 10 21:55:00.618576 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:00.618531    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:55:06.607424 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:06.607359    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:55:12.811396 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:12.811342    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-z5npz"]
Sep 10 21:55:22.029908 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:22.029875    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56fc979fdc-nwwdp_b2bb49ee-12c5-4d27-821d-62003ccf6724/copy-manifests/0.log"
Sep 10 21:55:22.046982 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:22.046941    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56fc979fdc-nwwdp_b2bb49ee-12c5-4d27-821d-62003ccf6724/manager/0.log"
Sep 10 21:55:22.133421 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:22.133373    2580 scope.go:117] "RemoveContainer" containerID="1d0938711e88bc5a6e7cdf032b15783b3821ea1b95c320a06fe39123ccf825e2"
Sep 10 21:55:22.617345 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:22.617312    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f95b84469-r6knn_146e35d0-7246-4987-afda-18eafbb93628/maas-api/0.log"
Sep 10 21:55:22.725598 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:22.725564    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817930-fb2dg_76a5957e-df05-46d9-ad32-79150fca3c8f/cleanup/0.log"
Sep 10 21:55:24.322384 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:24.322347    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-z5npz_5f352c56-671d-4097-b21e-cc931ff7398e/limitador/0.log"
Sep 10 21:55:24.447664 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:24.447628    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-t2gmk_c34ffe9d-afe1-4d32-bf40-3f4eacdd0423/manager/0.log"
Sep 10 21:55:25.137532 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:25.137495    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-745b97c954-n22zc_71ccef02-ed65-4e34-ac84-65ee182041d4/kube-auth-proxy/0.log"
Sep 10 21:55:26.508672 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:26.508638    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-klq8j_a4592aec-60bd-4e40-8b74-c38fa6c963b9/main/0.log"
Sep 10 21:55:26.620301 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:26.620256    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-ww5ks_58532a1e-d1d4-43e3-b156-349d7950668c/main/0.log"
Sep 10 21:55:32.339028 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.338984    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xhl4t/must-gather-d5vc9"]
Sep 10 21:55:32.339503 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.339396    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3" containerName="maas-api"
Sep 10 21:55:32.339503 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.339426    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3" containerName="maas-api"
Sep 10 21:55:32.339604 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.339506    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7cf37a5-50e4-4455-a1d2-7dcd50aea1c3" containerName="maas-api"
Sep 10 21:55:32.342762 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.342738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xhl4t/must-gather-d5vc9"
Sep 10 21:55:32.346600 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.346569    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xhl4t\"/\"kube-root-ca.crt\""
Sep 10 21:55:32.349460 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.349377    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xhl4t\"/\"default-dockercfg-z8wtj\""
Sep 10 21:55:32.349895 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.349866    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xhl4t\"/\"openshift-service-ca.crt\""
Sep 10 21:55:32.376256 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.376217    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xhl4t/must-gather-d5vc9"]
Sep 10 21:55:32.472909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.472863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlkqp\" (UniqueName: \"kubernetes.io/projected/3080d4b6-86cd-4afc-ba84-d9ee479b1666-kube-api-access-rlkqp\") pod \"must-gather-d5vc9\" (UID: \"3080d4b6-86cd-4afc-ba84-d9ee479b1666\") " pod="openshift-must-gather-xhl4t/must-gather-d5vc9"
Sep 10 21:55:32.472909 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.472908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3080d4b6-86cd-4afc-ba84-d9ee479b1666-must-gather-output\") pod \"must-gather-d5vc9\" (UID: \"3080d4b6-86cd-4afc-ba84-d9ee479b1666\") " pod="openshift-must-gather-xhl4t/must-gather-d5vc9"
Sep 10 21:55:32.574054 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.574010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlkqp\" (UniqueName: \"kubernetes.io/projected/3080d4b6-86cd-4afc-ba84-d9ee479b1666-kube-api-access-rlkqp\") pod \"must-gather-d5vc9\" (UID: \"3080d4b6-86cd-4afc-ba84-d9ee479b1666\") " pod="openshift-must-gather-xhl4t/must-gather-d5vc9"
Sep 10 21:55:32.574054 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.574054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3080d4b6-86cd-4afc-ba84-d9ee479b1666-must-gather-output\") pod \"must-gather-d5vc9\" (UID: \"3080d4b6-86cd-4afc-ba84-d9ee479b1666\") " pod="openshift-must-gather-xhl4t/must-gather-d5vc9"
Sep 10 21:55:32.574382 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.574362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3080d4b6-86cd-4afc-ba84-d9ee479b1666-must-gather-output\") pod \"must-gather-d5vc9\" (UID: \"3080d4b6-86cd-4afc-ba84-d9ee479b1666\") " pod="openshift-must-gather-xhl4t/must-gather-d5vc9"
Sep 10 21:55:32.595301 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.595232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlkqp\" (UniqueName: \"kubernetes.io/projected/3080d4b6-86cd-4afc-ba84-d9ee479b1666-kube-api-access-rlkqp\") pod \"must-gather-d5vc9\" (UID: \"3080d4b6-86cd-4afc-ba84-d9ee479b1666\") " pod="openshift-must-gather-xhl4t/must-gather-d5vc9"
Sep 10 21:55:32.653574 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.653533    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xhl4t/must-gather-d5vc9"
Sep 10 21:55:32.936160 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:32.936124    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xhl4t/must-gather-d5vc9"]
Sep 10 21:55:32.938912 ip-10-0-135-226 kubenswrapper[2580]: W0910 21:55:32.938881    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3080d4b6_86cd_4afc_ba84_d9ee479b1666.slice/crio-b0cf7aca80f024377816dcf95eac9586285789ea1e2fa89720b26542986ec6fa WatchSource:0}: Error finding container b0cf7aca80f024377816dcf95eac9586285789ea1e2fa89720b26542986ec6fa: Status 404 returned error can't find the container with id b0cf7aca80f024377816dcf95eac9586285789ea1e2fa89720b26542986ec6fa
Sep 10 21:55:33.576426 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:33.576365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xhl4t/must-gather-d5vc9" event={"ID":"3080d4b6-86cd-4afc-ba84-d9ee479b1666","Type":"ContainerStarted","Data":"b0cf7aca80f024377816dcf95eac9586285789ea1e2fa89720b26542986ec6fa"}
Sep 10 21:55:34.584363 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:34.584323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xhl4t/must-gather-d5vc9" event={"ID":"3080d4b6-86cd-4afc-ba84-d9ee479b1666","Type":"ContainerStarted","Data":"5f3e3ce6ed2d62c9cc7c7e1e1e82c2c1aae7f1acec6f0443f843b97398d2bbfe"}
Sep 10 21:55:34.584894 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:34.584379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xhl4t/must-gather-d5vc9" event={"ID":"3080d4b6-86cd-4afc-ba84-d9ee479b1666","Type":"ContainerStarted","Data":"d9db2d7776d2df25ca0154629f709b36e23c5eff8475a29a63f1334ab666fda8"}
Sep 10 21:55:34.601180 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:34.601101    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xhl4t/must-gather-d5vc9" podStartSLOduration=1.816751185 podStartE2EDuration="2.601073307s" totalImagesPullingTime="784.322122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:55:34.599531863 +0000 UTC m=+2173.247446032" watchObservedRunningTime="2026-09-10 21:55:34.601073307 +0000 UTC m=+2173.248987475"
Sep 10 21:55:35.339649 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:35.339609    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2r6cp_7e804440-819b-402d-ba84-f9f0c3ad4aeb/global-pull-secret-syncer/0.log"
Sep 10 21:55:35.538787 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:35.538741    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zs4b5_6d2f5710-7e64-49ab-a1c1-6d5cce892e5e/konnectivity-agent/0.log"
Sep 10 21:55:35.589337 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:35.589303    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-226.ec2.internal_2d6ceb36f4f0ad1940c1c73fe62eb8fd/haproxy/0.log"
Sep 10 21:55:40.397812 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:40.397772    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-z5npz_5f352c56-671d-4097-b21e-cc931ff7398e/limitador/0.log"
Sep 10 21:55:40.499089 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:40.499030    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-t2gmk_c34ffe9d-afe1-4d32-bf40-3f4eacdd0423/manager/0.log"
Sep 10 21:55:42.319070 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.319030    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rll9x_2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a/kube-state-metrics/0.log"
Sep 10 21:55:42.346039 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.346007    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rll9x_2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a/kube-rbac-proxy-main/0.log"
Sep 10 21:55:42.369992 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.369961    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rll9x_2dd8cc3d-bf22-4f4f-b40a-e9696a5d169a/kube-rbac-proxy-self/0.log"
Sep 10 21:55:42.401580 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.401489    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-79b8c94f99-j6jkn_d27eb360-a4ed-47cc-9bb3-84d8c3dbaeb8/metrics-server/0.log"
Sep 10 21:55:42.535229 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.535191    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hljtx_b6d3dc88-240a-4d69-8fcd-f244e3671ec6/node-exporter/0.log"
Sep 10 21:55:42.555083 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.555050    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hljtx_b6d3dc88-240a-4d69-8fcd-f244e3671ec6/kube-rbac-proxy/0.log"
Sep 10 21:55:42.578098 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.578003    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hljtx_b6d3dc88-240a-4d69-8fcd-f244e3671ec6/init-textfile/0.log"
Sep 10 21:55:42.757175 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.757137    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7c7dd78-3bee-420f-820e-5c433e7c4c54/prometheus/0.log"
Sep 10 21:55:42.779273 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.779237    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7c7dd78-3bee-420f-820e-5c433e7c4c54/config-reloader/0.log"
Sep 10 21:55:42.801684 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.801639    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7c7dd78-3bee-420f-820e-5c433e7c4c54/thanos-sidecar/0.log"
Sep 10 21:55:42.827800 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.827764    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7c7dd78-3bee-420f-820e-5c433e7c4c54/kube-rbac-proxy-web/0.log"
Sep 10 21:55:42.865843 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.865750    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7c7dd78-3bee-420f-820e-5c433e7c4c54/kube-rbac-proxy/0.log"
Sep 10 21:55:42.887104 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.887068    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7c7dd78-3bee-420f-820e-5c433e7c4c54/kube-rbac-proxy-thanos/0.log"
Sep 10 21:55:42.910846 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.910809    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7c7dd78-3bee-420f-820e-5c433e7c4c54/init-config-reloader/0.log"
Sep 10 21:55:42.978516 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.978475    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-xz2kd_86a2798c-6b0d-4f0e-afa2-0cf26a4288c8/prometheus-operator/0.log"
Sep 10 21:55:42.996350 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:42.996316    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-xz2kd_86a2798c-6b0d-4f0e-afa2-0cf26a4288c8/kube-rbac-proxy/0.log"
Sep 10 21:55:43.025856 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:43.025821    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-99zxv_24e2e7c0-a4eb-462e-b0dc-28d49941db9e/prometheus-operator-admission-webhook/0.log"
Sep 10 21:55:43.172256 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:43.172161    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54c9ff55c4-n7n74_2669f605-8f71-4e8e-97cd-de5a52a1405e/thanos-query/0.log"
Sep 10 21:55:43.202951 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:43.202912    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54c9ff55c4-n7n74_2669f605-8f71-4e8e-97cd-de5a52a1405e/kube-rbac-proxy-web/0.log"
Sep 10 21:55:43.229715 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:43.229680    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54c9ff55c4-n7n74_2669f605-8f71-4e8e-97cd-de5a52a1405e/kube-rbac-proxy/0.log"
Sep 10 21:55:43.272382 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:43.272354    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54c9ff55c4-n7n74_2669f605-8f71-4e8e-97cd-de5a52a1405e/prom-label-proxy/0.log"
Sep 10 21:55:43.306082 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:43.306052    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54c9ff55c4-n7n74_2669f605-8f71-4e8e-97cd-de5a52a1405e/kube-rbac-proxy-rules/0.log"
Sep 10 21:55:43.350883 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:43.350848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54c9ff55c4-n7n74_2669f605-8f71-4e8e-97cd-de5a52a1405e/kube-rbac-proxy-metrics/0.log"
Sep 10 21:55:46.285294 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.285254    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-5djjw_261a0890-0436-4126-be0d-16bfa97a14ae/download-server/0.log"
Sep 10 21:55:46.670508 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.670398    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"]
Sep 10 21:55:46.679606 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.679566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.683142 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.682879    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"]
Sep 10 21:55:46.834771 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.834722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-podres\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.835163 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.835140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-lib-modules\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.835321 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.835303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-proc\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.835463 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.835436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-sys\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.835600 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.835495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4bgc\" (UniqueName: \"kubernetes.io/projected/0b3fe217-b3d3-407c-b65e-dc84358b3776-kube-api-access-l4bgc\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.936848 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.936744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-lib-modules\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.937041 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.936970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-lib-modules\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.937041 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.937031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-proc\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.937176 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.937123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-sys\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.937237 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.937180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4bgc\" (UniqueName: \"kubernetes.io/projected/0b3fe217-b3d3-407c-b65e-dc84358b3776-kube-api-access-l4bgc\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.937298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.937242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-sys\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.937298 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.937272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-podres\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.937450 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.937387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-podres\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.937523 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.937177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b3fe217-b3d3-407c-b65e-dc84358b3776-proc\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:46.947046 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:46.947009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4bgc\" (UniqueName: \"kubernetes.io/projected/0b3fe217-b3d3-407c-b65e-dc84358b3776-kube-api-access-l4bgc\") pod \"perf-node-gather-daemonset-t8zpc\" (UID: \"0b3fe217-b3d3-407c-b65e-dc84358b3776\") " pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:47.000919 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:47.000879    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:47.293836 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:47.291051    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"]
Sep 10 21:55:47.696968 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:47.696921    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc" event={"ID":"0b3fe217-b3d3-407c-b65e-dc84358b3776","Type":"ContainerStarted","Data":"e3c47c3c7576c3ef4b92ce87152a3bb60400b40d9c010e4b6175d81e7bf8d24b"}
Sep 10 21:55:47.697196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:47.696998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc" event={"ID":"0b3fe217-b3d3-407c-b65e-dc84358b3776","Type":"ContainerStarted","Data":"8f6f3961efd6f021cf2822a6092a276356f1abd0431885536287672c9f9073c7"}
Sep 10 21:55:47.697196 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:47.697042    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:47.717578 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:47.717510    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc" podStartSLOduration=1.717492185 podStartE2EDuration="1.717492185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:55:47.715787196 +0000 UTC m=+2186.363701365" watchObservedRunningTime="2026-09-10 21:55:47.717492185 +0000 UTC m=+2186.365406353"
Sep 10 21:55:48.433097 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:48.433065    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s8nr2_f48024dc-9b26-46bf-8da8-b9ef5383bf45/dns/0.log"
Sep 10 21:55:48.459532 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:48.459496    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s8nr2_f48024dc-9b26-46bf-8da8-b9ef5383bf45/kube-rbac-proxy/0.log"
Sep 10 21:55:48.534606 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:48.534570    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zzrvw_6987485a-cc5d-45a1-a8e4-1eb4cf948766/dns-node-resolver/0.log"
Sep 10 21:55:49.078406 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:49.078368    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f9fbfb55-m5pd5_2f421fe3-e812-40cb-bfce-a734b0c1c5f8/registry/0.log"
Sep 10 21:55:49.101244 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:49.101211    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lj6vm_3f067fda-a406-4cf0-9135-b1efa82f373f/node-ca/0.log"
Sep 10 21:55:50.434154 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:50.434114    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-745b97c954-n22zc_71ccef02-ed65-4e34-ac84-65ee182041d4/kube-auth-proxy/0.log"
Sep 10 21:55:51.528460 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:51.528396    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qccrj_e9abb731-3053-4f97-ba96-f2b19f48bd3c/serve-healthcheck-canary/0.log"
Sep 10 21:55:52.206021 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:52.205988    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2zz2b_e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7/kube-rbac-proxy/0.log"
Sep 10 21:55:52.228996 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:52.228956    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2zz2b_e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7/exporter/0.log"
Sep 10 21:55:52.263984 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:52.263941    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2zz2b_e26b4fca-a631-4bc7-8a2c-bf9358d5d6c7/extractor/0.log"
Sep 10 21:55:53.714756 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:53.714719    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xhl4t/perf-node-gather-daemonset-t8zpc"
Sep 10 21:55:55.113704 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:55.113670    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56fc979fdc-nwwdp_b2bb49ee-12c5-4d27-821d-62003ccf6724/manager/0.log"
Sep 10 21:55:55.132874 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:55.132841    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56fc979fdc-nwwdp_b2bb49ee-12c5-4d27-821d-62003ccf6724/copy-manifests/0.log"
Sep 10 21:55:56.727874 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:56.727840    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5644d67b8d-24f8x_7b27662b-ebb7-4c4a-89db-39d6a85d017e/manager/0.log"
Sep 10 21:55:57.794141 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:57.794104    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-745b97c954-n22zc_71ccef02-ed65-4e34-ac84-65ee182041d4/kube-auth-proxy/0.log"
Sep 10 21:55:59.732109 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:55:59.732072    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-745b97c954-n22zc_71ccef02-ed65-4e34-ac84-65ee182041d4/kube-auth-proxy/0.log"
Sep 10 21:56:02.201005 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:02.200944    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-km762_5fa64996-d18c-4fe3-b347-8da4b600a85b/migrator/0.log"
Sep 10 21:56:02.219722 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:02.219686    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-km762_5fa64996-d18c-4fe3-b347-8da4b600a85b/graceful-termination/0.log"
Sep 10 21:56:03.766531 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:03.766502    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxt6_ed0bb520-aceb-44a9-8796-178e661d1482/kube-multus-additional-cni-plugins/0.log"
Sep 10 21:56:03.805568 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:03.805536    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxt6_ed0bb520-aceb-44a9-8796-178e661d1482/egress-router-binary-copy/0.log"
Sep 10 21:56:03.850492 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:03.850446    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxt6_ed0bb520-aceb-44a9-8796-178e661d1482/cni-plugins/0.log"
Sep 10 21:56:03.890577 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:03.890541    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxt6_ed0bb520-aceb-44a9-8796-178e661d1482/bond-cni-plugin/0.log"
Sep 10 21:56:03.928634 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:03.928600    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxt6_ed0bb520-aceb-44a9-8796-178e661d1482/routeoverride-cni/0.log"
Sep 10 21:56:03.966518 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:03.966478    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxt6_ed0bb520-aceb-44a9-8796-178e661d1482/whereabouts-cni-bincopy/0.log"
Sep 10 21:56:04.008296 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:04.008262    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxt6_ed0bb520-aceb-44a9-8796-178e661d1482/whereabouts-cni/0.log"
Sep 10 21:56:04.611082 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:04.610985    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-m2mjz_1e48ee8f-64c3-4ab2-950c-f09d5043d472/kube-multus/0.log"
Sep 10 21:56:04.749986 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:04.749942    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kgjnr_51decde2-f81d-4c6f-b857-8fd6150f288a/network-metrics-daemon/0.log"
Sep 10 21:56:04.776300 ip-10-0-135-226 kubenswrapper[2580]: I0910 21:56:04.776263    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kgjnr_51decde2-f81d-4c6f-b857-8fd6150f288a/kube-rbac-proxy/0.log"