Sep 11 14:38:08.443759 ip-10-0-128-223 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 14:38:08.443770 ip-10-0-128-223 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 14:38:08.443780 ip-10-0-128-223 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 14:38:08.444200 ip-10-0-128-223 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 14:38:18.615110 ip-10-0-128-223 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 14:38:18.615127 ip-10-0-128-223 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5a5eb48c1be54237899928edfb3bb339 --
Sep 11 14:39:52.220017 ip-10-0-128-223 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 14:39:52.640954 ip-10-0-128-223 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 14:39:52.640954 ip-10-0-128-223 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 14:39:52.640954 ip-10-0-128-223 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 14:39:52.640954 ip-10-0-128-223 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 14:39:52.640954 ip-10-0-128-223 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 14:39:52.642252 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.642165    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 14:39:52.646441 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646426    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:39:52.646441 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646443    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646447    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646451    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646454    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646457    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646460    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646463    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646465    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646468    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646472    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646475    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646478    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646482    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646486    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646489    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646493    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646495    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646498    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646501    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:39:52.646507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646504    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646507    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646510    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646513    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646517    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646520    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646524    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646527    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646530    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646533    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646537    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646539    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646542    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646544    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646547    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646549    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646552    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646555    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646558    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:39:52.646988 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646560    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646563    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646566    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646568    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646571    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646573    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646576    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646579    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646581    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646584    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646586    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646589    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646591    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646594    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646596    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646599    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646602    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646605    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646607    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646610    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:39:52.647485 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646613    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646615    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646619    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646623    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646626    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646628    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646631    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646633    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646636    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646638    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646641    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646643    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646646    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646649    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646651    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646655    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646658    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646660    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646663    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646665    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:39:52.647991 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646667    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646670    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646672    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646675    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646677    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646680    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646683    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646686    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646689    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646693    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646695    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646698    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646700    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646703    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646705    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646708    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646710    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646713    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646727    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.646730    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:39:52.648506 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647410    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647417    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647419    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647422    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647426    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647429    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647432    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647434    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647439    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647443    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647446    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647449    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647452    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647455    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647457    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647460    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647463    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647465    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647468    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647472    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:39:52.648997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647475    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647477    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647480    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647482    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647485    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647488    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647491    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647495    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647497    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647500    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647503    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647505    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647508    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647510    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647513    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647515    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647519    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647522    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647526    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647528    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:39:52.649512 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647531    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647534    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647537    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647540    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647543    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647545    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647548    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647551    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647553    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647556    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647559    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647561    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647564    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647567    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647569    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647572    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647574    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647577    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647579    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:39:52.650043 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647582    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647585    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647587    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647589    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647592    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647594    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647597    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647599    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647602    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647607    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647609    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647612    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647615    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647617    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647621    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647625    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647628    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647632    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647635    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:39:52.650507 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647637    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647640    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647643    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647645    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647648    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647650    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647653    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647656    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647658    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647661    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647664    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647667    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647669    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647672    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647674    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647677    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647679    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647682    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647684    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647687    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:39:52.650994 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.647689    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648050    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648060    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648063    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648066    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648071    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648074    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648077    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648080    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648082    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648085    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648088    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648090    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648093    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648095    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648098    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648101    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648103    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648106    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648109    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648112    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:39:52.651579 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648115    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648117    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648120    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648123    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648125    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648128    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648130    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648133    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648135    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648138    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648142    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648145    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648147    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648151    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648154    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648156    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648159    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648162    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648164    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:39:52.652114 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648167    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648169    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648175    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648178    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648181    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648184    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648186    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648189    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648192    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648194    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648197    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648200    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648202    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648205    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648207    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648210    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648213    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648216    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648220    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:39:52.652573 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648223    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648226    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648229    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648232    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648235    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648237    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648240    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648244    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648246    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648249    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648251    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648254    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648256    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648259    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648262    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648265    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648267    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648270    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648272    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:39:52.653127 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648275    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648278    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648280    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648283    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648285    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648288    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648290    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648293    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648297    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648299    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648302    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648304    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648307    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648309    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648312    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648314    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648317    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648319    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648322    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648324    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:39:52.653595 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648328    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.648331    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648408    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648440    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648446    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648452    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648457    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648461    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648466    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648470    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648473    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648477    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648480    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648483    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648487    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648490    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648493    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648496    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648499    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648502    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648506    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648510    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648513    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648516    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 14:39:52.654139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648520    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648524    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648527    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648530    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648533    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648536    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648539    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648542    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648546    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648549    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648553    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648556    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648559    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648563    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648566    2559 flags.go:64] FLAG: --enable-server="true"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648569    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648574    2559 flags.go:64] FLAG: --event-burst="100"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648577    2559 flags.go:64] FLAG: --event-qps="50"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648579    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648582    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648585    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648589    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648593    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648596    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648599    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 11 14:39:52.654705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648602    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648605    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648607    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648610    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648613    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648616    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648619    2559 flags.go:64] FLAG: --feature-gates=""
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648623    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648626    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648629    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648632    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648635    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648638    2559 flags.go:64] FLAG: --help="false"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648641    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-128-223.ec2.internal"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648645    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648648    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648651    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648654    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648658    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648661    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648665    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648668    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648671    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 14:39:52.655332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648675    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648678    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648681    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648683    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648686    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648690    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648692    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648695    2559 flags.go:64] FLAG: --lock-file=""
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648698    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648701    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648704    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648710    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648712    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648729    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648732    2559 flags.go:64] FLAG: --logging-format="text"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648735    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648738    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648741    2559 flags.go:64] FLAG: --manifest-url=""
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648745    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648749    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648752    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648756    2559 flags.go:64] FLAG: --max-pods="110"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648760    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648763    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648766    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 14:39:52.655962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648769    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648772    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648775    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648778    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648787    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648790    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648793    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648796    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648800    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648805    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648808    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648812    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648815    2559 flags.go:64] FLAG: --port="10250"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648818    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648821    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09929b4a654f0f8b7"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648824    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648827    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648830    2559 flags.go:64] FLAG: --register-node="true"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648833    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648837    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648840    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648843    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648846    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648849    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648852    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 14:39:52.656568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648855    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648858    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648861    2559 flags.go:64] FLAG: --runonce="false"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648864    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648867    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648870    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648873    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648876    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648879    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648882    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648885    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648889    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648892    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648895    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648898    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648901    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648904    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648907    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648912    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648915    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648918    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648922    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648925    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648928    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648930    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 14:39:52.657200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648933    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648936    2559 flags.go:64] FLAG: --v="2"
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648940    2559 flags.go:64] FLAG: --version="false"
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648944    2559 flags.go:64] FLAG: --vmodule=""
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648947    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.648950    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649096    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649110    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649114    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649118    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649121    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649124    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649127    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649130    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649133    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649135    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649138    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649141    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649144    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649147    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649149    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:39:52.657890 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649152    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649155    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649157    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649160    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649163    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649166    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649168    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649171    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649174    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649177    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649179    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649182    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649184    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649187    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649191    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649194    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649196    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649199    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649206    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649209    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:39:52.658409 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649212    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649215    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649217    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649220    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649223    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649226    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649229    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649232    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649234    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649237    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649240    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649243    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649245    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649248    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649250    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649253    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649255    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649258    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649261    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649263    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:39:52.658926 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649266    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649271    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649275    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649278    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649280    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649283    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649287    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649289    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649292    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649295    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649300    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649305    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649308    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649311    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649314    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649317    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649320    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649322    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649325    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:39:52.659408 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649328    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649331    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649334    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649336    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649339    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649341    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649344    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649347    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649349    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649352    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649355    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649357    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649360    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649362    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649365    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649368    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649371    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649373    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649377    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649380    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:39:52.659900 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649383    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649386    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649388    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649392    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.649395    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.649401    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.657006    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.657022    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.657871    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.657879    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657935    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657941    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657945    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657948    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657951    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:39:52.660384 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657955    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657959    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657963    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657966    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657969    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657973    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657975    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657978    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657981    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657984    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657986    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657989    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657993    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657996    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.657999    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658002    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658004    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658007    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658010    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:39:52.660777 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658013    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658015    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658018    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658021    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658024    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658028    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658030    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658033    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658036    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658038    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658041    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658044    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658046    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658049    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658051    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658054    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658056    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658059    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658062    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658065    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:39:52.661317 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658067    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658070    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658073    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658075    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658078    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658081    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658083    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658086    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658088    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658091    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658093    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658096    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658099    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658101    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658104    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658106    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658110    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658113    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658116    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658118    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:39:52.661837 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658121    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658124    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658126    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658129    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658132    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658134    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658138    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658142    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658144    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658148    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658150    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658153    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658156    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658159    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658162    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658164    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658167    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658169    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658172    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:39:52.662334 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658174    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658177    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658180    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658182    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658185    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658187    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658190    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658193    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658196    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658199    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658203    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658206    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658209    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658211    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658214    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:39:52.662820 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658217    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.658222    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658334    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658344    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658348    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658350    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658354    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658357    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658360    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658362    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658365    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658368    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658370    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658373    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658375    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:39:52.663189 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658378    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658381    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658383    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658386    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658389    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658391    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658395    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658399    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658402    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658405    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658408    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658411    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658414    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658419    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658423    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658426    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658429    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658432    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658434    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:39:52.663567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658437    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658440    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658443    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658445    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658448    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658451    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658453    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658456    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658459    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658461    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658464    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658466    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658469    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658472    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658474    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658477    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658479    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658482    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658485    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658487    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:39:52.664055 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658490    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658492    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658495    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658497    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658500    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658502    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658506    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658509    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658511    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658514    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658517    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658520    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658523    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658526    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658528    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658531    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658534    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658536    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658539    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:39:52.664547 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658541    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658544    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658546    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658549    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658552    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658554    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658557    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658559    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658562    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658565    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658567    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658569    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658572    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658576    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658579    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658582    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658585    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658588    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658591    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:39:52.665013 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658594    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658597    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658600    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658603    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658606    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658608    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658611    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658613    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:52.658616    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.658622    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.658736    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.661051    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.662164    2559 server.go:1019] "Starting client certificate rotation"
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.662260    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 14:39:52.665510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.662293    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 14:39:52.665878 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.665095    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 14:39:52.670406 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.670384    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 14:39:52.680069 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.680051    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 11 14:39:52.683541 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.683527    2559 log.go:25] "Validated CRI v1 image API"
Sep 11 14:39:52.684355 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.684337    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 14:39:52.689491 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.689468    2559 fs.go:135] Filesystem UUIDs: map[6ba1ab09-c65f-4689-8fab-58b7487b227c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 980ef3ab-126c-4215-b8f1-593d39fddfbd:/dev/nvme0n1p3]
Sep 11 14:39:52.689551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.689491    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 14:39:52.694558 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.694443    2559 manager.go:216] Machine: {Timestamp:2026-09-11 14:39:52.693019701 +0000 UTC m=+0.369383026 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099894 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bfd2bbaade5f0f07511175fa1c2ca SystemUUID:ec2bfd2b-baad-e5f0-f075-11175fa1c2ca BootID:5a5eb48c-1be5-4237-8999-28edfb3bb339 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:83:e7:48:b2:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:83:e7:48:b2:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:15:eb:5f:12:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 14:39:52.694558 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.694552    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 14:39:52.694671 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.694627    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 14:39:52.698147 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.698129    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 14:39:52.698394 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.698361    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 14:39:52.698530 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.698397    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-223.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 14:39:52.699187 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.699177    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 14:39:52.699219 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.699189    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 14:39:52.699219 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.699197    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 14:39:52.700923 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.700910    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 14:39:52.702145 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.702135    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 14:39:52.706741 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.706730    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 14:39:52.706860 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.706850    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 14:39:52.706892 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.706880    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 14:39:52.706925 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.706892    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 14:39:52.706925 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.706902    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 14:39:52.706925 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.706913    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 11 14:39:52.706925 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.706921    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 14:39:52.707385 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.707372    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 14:39:52.707438 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.707391    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 14:39:52.710415 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.710389    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 14:39:52.712728 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.712703    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 14:39:52.712795 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.712753    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 14:39:52.714863 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714849    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 14:39:52.714863 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714867    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 14:39:52.714863 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714873    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 14:39:52.715035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714878    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 14:39:52.715035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714887    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 14:39:52.715035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714892    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 14:39:52.715035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714905    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 14:39:52.715035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714911    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 14:39:52.715035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714918    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 14:39:52.715035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714924    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 14:39:52.715035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714932    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 14:39:52.715035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.714937    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 14:39:52.716935 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.716910    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 14:39:52.717052 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.717040    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 14:39:52.720513 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.720498    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-223.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 14:39:52.720710 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.720688    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 14:39:52.720810 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.720788    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 14:39:52.721350 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.721337    2559 server.go:1268] "Started kubelet"
Sep 11 14:39:52.721420 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.721393    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 14:39:52.721495 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.721465    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 14:39:52.721540 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.721518    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 14:39:52.722260 ip-10-0-128-223 systemd[1]: Started Kubernetes Kubelet.
Sep 11 14:39:52.723602 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.723445    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 14:39:52.723707 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.723502    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 14:39:52.726248 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.726224    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 14:39:52.729484 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.729467    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 14:39:52.730227 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.730213    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 14:39:52.730311 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.730295    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:52.730695 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.729486    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-223.ec2.internal.18d44b294c735a67  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-223.ec2.internal,UID:ip-10-0-128-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-223.ec2.internal,},FirstTimestamp:2026-09-11 14:39:52.721300071 +0000 UTC m=+0.397663399,LastTimestamp:2026-09-11 14:39:52.721300071 +0000 UTC m=+0.397663399,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-223.ec2.internal,}"
Sep 11 14:39:52.730973 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.730953    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 14:39:52.731067 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.730952    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 14:39:52.731154 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.731071    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 14:39:52.731154 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.730975    2559 factory.go:55] Registering systemd factory
Sep 11 14:39:52.731154 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.731148    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 11 14:39:52.731291 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.731156    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 14:39:52.731291 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.731164    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 14:39:52.731355 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.731349    2559 factory.go:153] Registering CRI-O factory
Sep 11 14:39:52.731386 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.731362    2559 factory.go:223] Registration of the crio container factory successfully
Sep 11 14:39:52.731418 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.731411    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 14:39:52.731447 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.731427    2559 factory.go:103] Registering Raw factory
Sep 11 14:39:52.731447 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.731437    2559 manager.go:1196] Started watching for new ooms in manager
Sep 11 14:39:52.731936 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.731917    2559 manager.go:349] Starting recovery of all containers
Sep 11 14:39:52.740268 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.740239    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 14:39:52.740363 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.740278    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 14:39:52.746219 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.746200    2559 manager.go:354] Recovery completed
Sep 11 14:39:52.751687 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.751675    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:39:52.753204 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.753188    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:39:52.753260 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.753216    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:39:52.753260 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.753226    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:39:52.753850 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.753833    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 14:39:52.753850 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.753849    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 14:39:52.753958 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.753852    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ct5gb"
Sep 11 14:39:52.753958 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.753857    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:39:52.753958 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.753867    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 14:39:52.754602 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.754558    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:39:52.754665 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.754615    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:39:52.754665 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.754626    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:39:52.755346 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.755252    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-223.ec2.internal.18d44b294e5a2936  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-223.ec2.internal,UID:ip-10-0-128-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-223.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-223.ec2.internal,},FirstTimestamp:2026-09-11 14:39:52.75320351 +0000 UTC m=+0.429566836,LastTimestamp:2026-09-11 14:39:52.75320351 +0000 UTC m=+0.429566836,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-223.ec2.internal,}"
Sep 11 14:39:52.755964 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.755950    2559 policy_none.go:49] "None policy: Start"
Sep 11 14:39:52.756014 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.755969    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 14:39:52.756014 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.755983    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 14:39:52.757888 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.757877    2559 policy_none.go:47] "Start"
Sep 11 14:39:52.762483 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.762467    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ct5gb"
Sep 11 14:39:52.763821 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.763757    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-223.ec2.internal.18d44b294e5a6b79  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-223.ec2.internal,UID:ip-10-0-128-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-223.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-223.ec2.internal,},FirstTimestamp:2026-09-11 14:39:52.753220473 +0000 UTC m=+0.429583799,LastTimestamp:2026-09-11 14:39:52.753220473 +0000 UTC m=+0.429583799,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-223.ec2.internal,}"
Sep 11 14:39:52.767421 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.767362    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-223.ec2.internal.18d44b294e5a8e87  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-223.ec2.internal,UID:ip-10-0-128-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-223.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-223.ec2.internal,},FirstTimestamp:2026-09-11 14:39:52.753229447 +0000 UTC m=+0.429592773,LastTimestamp:2026-09-11 14:39:52.753229447 +0000 UTC m=+0.429592773,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-223.ec2.internal,}"
Sep 11 14:39:52.784281 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.784265    2559 manager.go:337] "Starting Device Plugin manager"
Sep 11 14:39:52.804449 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.784315    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 14:39:52.804449 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.784328    2559 server.go:87] "Starting device plugin registration server"
Sep 11 14:39:52.804449 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.784611    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 14:39:52.804449 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.784622    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 14:39:52.804449 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.784713    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 14:39:52.804449 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.784807    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 14:39:52.804449 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.784816    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 14:39:52.804449 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.785326    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 14:39:52.804449 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.785378    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:52.818469 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.818447    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 14:39:52.819577 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.819558    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 14:39:52.819671 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.819606    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 14:39:52.819671 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.819626    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 14:39:52.819671 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.819667    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 14:39:52.822680 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.822663    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:39:52.885404 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.885372    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:39:52.886465 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.886446    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:39:52.886566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.886478    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:39:52.886566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.886492    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:39:52.886566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.886514    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-223.ec2.internal"
Sep 11 14:39:52.896799 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.896743    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-223.ec2.internal"
Sep 11 14:39:52.896799 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.896767    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-223.ec2.internal\": node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:52.920546 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.920513    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-223.ec2.internal"]
Sep 11 14:39:52.920637 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.920584    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:39:52.921948 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.921929    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:39:52.922015 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.921959    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:39:52.922015 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.921969    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:39:52.924113 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.924102    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:39:52.924230 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.924217    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:52.924267 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.924247    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:39:52.924982 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.924965    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:39:52.925045 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.924989    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:39:52.925045 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.924969    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:39:52.925045 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.925030    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:39:52.925045 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.925002    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:39:52.925230 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.925045    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:39:52.927389 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.927370    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:52.927537 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.927395    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:39:52.928606 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.928589    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:39:52.928689 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.928616    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:39:52.928689 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.928629    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:39:52.930841 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.930699    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:52.932513 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.932490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73cb3b572770e472576c1c3acbe52a84-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal\" (UID: \"73cb3b572770e472576c1c3acbe52a84\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:52.932565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.932522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73cb3b572770e472576c1c3acbe52a84-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal\" (UID: \"73cb3b572770e472576c1c3acbe52a84\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:52.932565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:52.932538    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b26ed7e8c79abb7d6bd735b73ba0b5d9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-223.ec2.internal\" (UID: \"b26ed7e8c79abb7d6bd735b73ba0b5d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:52.952297 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.952275    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-223.ec2.internal\" not found" node="ip-10-0-128-223.ec2.internal"
Sep 11 14:39:52.956562 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:52.956547    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-223.ec2.internal\" not found" node="ip-10-0-128-223.ec2.internal"
Sep 11 14:39:53.030953 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:53.030934    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:53.033115 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:53.033102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73cb3b572770e472576c1c3acbe52a84-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal\" (UID: \"73cb3b572770e472576c1c3acbe52a84\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:53.033162 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:53.033124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73cb3b572770e472576c1c3acbe52a84-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal\" (UID: \"73cb3b572770e472576c1c3acbe52a84\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:53.033162 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:53.033141    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b26ed7e8c79abb7d6bd735b73ba0b5d9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-223.ec2.internal\" (UID: \"b26ed7e8c79abb7d6bd735b73ba0b5d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:53.033226 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:53.033182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b26ed7e8c79abb7d6bd735b73ba0b5d9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-223.ec2.internal\" (UID: \"b26ed7e8c79abb7d6bd735b73ba0b5d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:53.033226 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:53.033189    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73cb3b572770e472576c1c3acbe52a84-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal\" (UID: \"73cb3b572770e472576c1c3acbe52a84\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:53.033226 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:53.033213    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73cb3b572770e472576c1c3acbe52a84-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal\" (UID: \"73cb3b572770e472576c1c3acbe52a84\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:53.131351 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:53.131312    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:53.232041 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:53.231985    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:53.254307 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:53.254287    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:53.259154 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:53.259077    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:53.333009 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:53.332978    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:54.787198 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:54.787172    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:54.887781 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:54.887722    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:54.988358 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:54.988327    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:55.015643 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.015619    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 14:39:55.016023 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.015755    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 14:39:55.088928 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:55.088903    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:55.122347 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.121968    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 14:34:54 +0000 UTC" deadline="2028-03-17 09:57:49.112644231 +0000 UTC"
Sep 11 14:39:55.122347 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.122346    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13267h17m53.990302742s"
Sep 11 14:39:55.189326 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:55.189263    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:55.193567 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.193549    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:39:55.194625 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.194608    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:39:55.194705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.194638    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:39:55.194705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.194648    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:39:55.290196 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:55.290169    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:55.378931 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:55.378785    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cb3b572770e472576c1c3acbe52a84.slice/crio-9b4d18ff63cf3e040b185f3842557aadfa280c2664097f5697eb2f7c054b2c34 WatchSource:0}: Error finding container 9b4d18ff63cf3e040b185f3842557aadfa280c2664097f5697eb2f7c054b2c34: Status 404 returned error can't find the container with id 9b4d18ff63cf3e040b185f3842557aadfa280c2664097f5697eb2f7c054b2c34
Sep 11 14:39:55.379115 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:55.379102    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26ed7e8c79abb7d6bd735b73ba0b5d9.slice/crio-e37582bfc03beb215ae6b7ee0b104209251b5a2e5ad2b259021adafa48ae9746 WatchSource:0}: Error finding container e37582bfc03beb215ae6b7ee0b104209251b5a2e5ad2b259021adafa48ae9746: Status 404 returned error can't find the container with id e37582bfc03beb215ae6b7ee0b104209251b5a2e5ad2b259021adafa48ae9746
Sep 11 14:39:55.383216 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.383199    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:39:55.391027 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:55.391003    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-223.ec2.internal\" not found"
Sep 11 14:39:55.452743 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.452690    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:39:55.484204 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.484181    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:55.486142 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.486129    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:39:55.494939 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.494916    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 14:39:55.496850 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.496836    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal"
Sep 11 14:39:55.511031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.511011    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 14:39:55.633945 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.633911    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:39:55.898843 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:55.898817    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:39:56.061744 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.061669    2559 apiserver.go:52] "Watching apiserver"
Sep 11 14:39:56.072036 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.072009    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 14:39:56.073233 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.073205    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-2gmn2","openshift-network-diagnostics/network-check-target-9l6tg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf","openshift-cluster-node-tuning-operator/tuned-5bk6j","openshift-dns/node-resolver-7lxrw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal","openshift-multus/multus-additional-cni-plugins-8r5kp","openshift-multus/network-metrics-daemon-2snh4","openshift-network-operator/iptables-alerter-9qb5m","openshift-ovn-kubernetes/ovnkube-node-d6xtk","kube-system/global-pull-secret-syncer-24976","kube-system/konnectivity-agent-rm7gp","kube-system/kube-apiserver-proxy-ip-10-0-128-223.ec2.internal","openshift-image-registry/node-ca-6r78k"]
Sep 11 14:39:56.075900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.075772    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.078058 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.078040    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:56.078149 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.078115    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:39:56.079211 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.079024    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 14:39:56.079431 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.079409    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 14:39:56.079984 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.079890    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 14:39:56.079984 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.079905    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 14:39:56.079984 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.079962    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 14:39:56.080192 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.080166    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.080248 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.080208    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-464jn\""
Sep 11 14:39:56.082362 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.082338    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.082805 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.082783    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 14:39:56.082903 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.082825    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 14:39:56.085640 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.084252    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gdc62\""
Sep 11 14:39:56.085640 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.084471    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 14:39:56.085640 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.084689    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 14:39:56.086365 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.086346    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:39:56.086584 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.086565    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wmmh4\""
Sep 11 14:39:56.086683 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.086647    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 14:39:56.087947 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.087766    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7lxrw"
Sep 11 14:39:56.090613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.090407    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 14:39:56.090613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.090478    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 14:39:56.090613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.090407    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-29mf9\""
Sep 11 14:39:56.091382 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.091366    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6r78k"
Sep 11 14:39:56.093929 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.093910    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9qb5m"
Sep 11 14:39:56.094611 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094590    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-modprobe-d\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.094611 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-sys\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.094779 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6bb6b12f-ef56-49af-a24e-9759831b31b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.094779 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094678    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-device-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.094779 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094701    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-etc-selinux\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.094779 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094708    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 14:39:56.094779 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094721    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q9nd6\""
Sep 11 14:39:56.095018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-sys-fs\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.095018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094836    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-systemd\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqpg\" (UniqueName: \"kubernetes.io/projected/63d954c0-13c8-4f70-bf69-ce8c999c60ea-kube-api-access-wxqpg\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b7d0ed0-8337-4b8c-adfa-4918f5150593-hosts-file\") pod \"node-resolver-7lxrw\" (UID: \"7b7d0ed0-8337-4b8c-adfa-4918f5150593\") " pod="openshift-dns/node-resolver-7lxrw"
Sep 11 14:39:56.095018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094901    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-registration-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.095018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094907    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 14:39:56.095018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-sysctl-d\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094943    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-lib-modules\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.094965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-var-lib-kubelet\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095003    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 14:39:56.095018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-dbus\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095047    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-socket-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095069    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-sysctl-conf\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095090    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-host\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095111    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63d954c0-13c8-4f70-bf69-ce8c999c60ea-tmp\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095133    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b7d0ed0-8337-4b8c-adfa-4918f5150593-tmp-dir\") pod \"node-resolver-7lxrw\" (UID: \"7b7d0ed0-8337-4b8c-adfa-4918f5150593\") " pod="openshift-dns/node-resolver-7lxrw"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095166    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n64r\" (UniqueName: \"kubernetes.io/projected/6bb6b12f-ef56-49af-a24e-9759831b31b7-kube-api-access-5n64r\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-sysconfig\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095215    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-tuned\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-kubernetes\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095275    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bb6b12f-ef56-49af-a24e-9759831b31b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095319    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095346    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2sfl\" (UniqueName: \"kubernetes.io/projected/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-kube-api-access-t2sfl\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-run\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095393    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-kubelet-config\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095415    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7zr7\" (UniqueName: \"kubernetes.io/projected/7b7d0ed0-8337-4b8c-adfa-4918f5150593-kube-api-access-t7zr7\") pod \"node-resolver-7lxrw\" (UID: \"7b7d0ed0-8337-4b8c-adfa-4918f5150593\") " pod="openshift-dns/node-resolver-7lxrw"
Sep 11 14:39:56.095566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095448    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bb6b12f-ef56-49af-a24e-9759831b31b7-system-cni-dir\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.096355 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095547    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bb6b12f-ef56-49af-a24e-9759831b31b7-cnibin\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.096355 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095573    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bb6b12f-ef56-49af-a24e-9759831b31b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.096355 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095607    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6bb6b12f-ef56-49af-a24e-9759831b31b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.096355 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.095633    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:56.097117 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.096880    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:39:56.097117 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.096944    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:39:56.098053 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.097788    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 14:39:56.098053 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.097846    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 14:39:56.098053 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.097797    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:39:56.098703 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.098687    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4vbsc\""
Sep 11 14:39:56.099191 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.099174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:39:56.099267 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.099236    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:39:56.102312 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.102296    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.102860 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.102823    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 14:39:56.105079 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.105059    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.106529 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.106509    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 14:39:56.106664 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.106629    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 14:39:56.107270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.107245    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:39:56.107517 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.107495    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 14:39:56.108096 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.107971    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 14:39:56.108096 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.108017    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cwmpv\""
Sep 11 14:39:56.108272 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.107951    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 14:39:56.108847 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.108415    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ddcg5\""
Sep 11 14:39:56.108847 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.108449    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 14:39:56.108847 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.108490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 14:39:56.109578 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.109543    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 14:39:56.110172 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.110142    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dn9nk\""
Sep 11 14:39:56.110249 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.110212    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 14:39:56.131428 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.131410    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rdlnr"
Sep 11 14:39:56.148673 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.148651    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rdlnr"
Sep 11 14:39:56.177920 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.177872    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal" event={"ID":"73cb3b572770e472576c1c3acbe52a84","Type":"ContainerStarted","Data":"9b4d18ff63cf3e040b185f3842557aadfa280c2664097f5697eb2f7c054b2c34"}
Sep 11 14:39:56.179252 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.179227    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-223.ec2.internal" event={"ID":"b26ed7e8c79abb7d6bd735b73ba0b5d9","Type":"ContainerStarted","Data":"e37582bfc03beb215ae6b7ee0b104209251b5a2e5ad2b259021adafa48ae9746"}
Sep 11 14:39:56.186288 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.186271    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bb6b12f-ef56-49af-a24e-9759831b31b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196254    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-slash\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-modprobe-d\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.196472    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196508    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-modprobe-d\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.196538    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret podName:c89e9a20-c4e4-4c03-9601-8e47f2ad2df2 nodeName:}" failed. No retries permitted until 2026-09-11 14:39:56.696516196 +0000 UTC m=+3.019144567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret") pod "global-pull-secret-syncer-24976" (UID: "c89e9a20-c4e4-4c03-9601-8e47f2ad2df2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196558    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-sys\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-log-socket\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-conf-dir\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sv72\" (UniqueName: \"kubernetes.io/projected/a8003a60-3e02-4e75-b168-c540a3975f4d-kube-api-access-9sv72\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196660    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-systemd\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196682    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqpg\" (UniqueName: \"kubernetes.io/projected/63d954c0-13c8-4f70-bf69-ce8c999c60ea-kube-api-access-wxqpg\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-run-netns\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9533b2d-a936-407d-ad6f-e3480bcdd109-serviceca\") pod \"node-ca-6r78k\" (UID: \"c9533b2d-a936-407d-ad6f-e3480bcdd109\") " pod="openshift-image-registry/node-ca-6r78k"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-registration-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.197419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-sysctl-d\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-sys\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196797    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-lib-modules\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196858    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-cni-bin\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196881    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bb6b12f-ef56-49af-a24e-9759831b31b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-run-k8s-cni-cncf-io\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-sysctl-conf\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-host\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63d954c0-13c8-4f70-bf69-ce8c999c60ea-tmp\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196992    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-registration-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.196996    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197025    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-kubelet\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197046    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-host\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197078    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-daemon-config\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-etc-kubernetes\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-sysconfig\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.198327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-sysctl-d\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197186    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-cnibin\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-socket-dir-parent\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197249    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-run-multus-certs\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197273    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczgc\" (UniqueName: \"kubernetes.io/projected/c9533b2d-a936-407d-ad6f-e3480bcdd109-kube-api-access-fczgc\") pod \"node-ca-6r78k\" (UID: \"c9533b2d-a936-407d-ad6f-e3480bcdd109\") " pod="openshift-image-registry/node-ca-6r78k"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197300    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-kubernetes\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197325    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bb6b12f-ef56-49af-a24e-9759831b31b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-sysctl-conf\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197350    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zc5q\" (UniqueName: \"kubernetes.io/projected/fde006ea-2cba-4b23-b83e-6f7703193a31-kube-api-access-8zc5q\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197374    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvm47\" (UniqueName: \"kubernetes.io/projected/721c30dc-143f-4729-883a-12be9fd78201-kube-api-access-bvm47\") pod \"iptables-alerter-9qb5m\" (UID: \"721c30dc-143f-4729-883a-12be9fd78201\") " pod="openshift-network-operator/iptables-alerter-9qb5m"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-run-openvswitch\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/84d7fdb2-524e-4627-8740-17ba2dcfefa0-agent-certs\") pod \"konnectivity-agent-rm7gp\" (UID: \"84d7fdb2-524e-4627-8740-17ba2dcfefa0\") " pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-lib-modules\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197553    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197559    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-systemd\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197608    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-sysconfig\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.199150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197659    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-kubernetes\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197645    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197708    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-kubelet-config\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-run-netns\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197785    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc23afba-cb1c-4664-badc-e76289c18e6d-env-overrides\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197809    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc23afba-cb1c-4664-badc-e76289c18e6d-ovn-node-metrics-cert\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197837    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7zr7\" (UniqueName: \"kubernetes.io/projected/7b7d0ed0-8337-4b8c-adfa-4918f5150593-kube-api-access-t7zr7\") pod \"node-resolver-7lxrw\" (UID: \"7b7d0ed0-8337-4b8c-adfa-4918f5150593\") " pod="openshift-dns/node-resolver-7lxrw"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bb6b12f-ef56-49af-a24e-9759831b31b7-system-cni-dir\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bb6b12f-ef56-49af-a24e-9759831b31b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bb6b12f-ef56-49af-a24e-9759831b31b7-cnibin\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6bb6b12f-ef56-49af-a24e-9759831b31b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89nm9\" (UniqueName: \"kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9\") pod \"network-check-target-9l6tg\" (UID: \"4bd24af6-f410-4833-816b-772fa4da84f1\") " pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bb6b12f-ef56-49af-a24e-9759831b31b7-system-cni-dir\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-systemd-units\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.197981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-kubelet-config\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bb6b12f-ef56-49af-a24e-9759831b31b7-cnibin\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198192    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8003a60-3e02-4e75-b168-c540a3975f4d-cni-binary-copy\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.200031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198232    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9533b2d-a936-407d-ad6f-e3480bcdd109-host\") pod \"node-ca-6r78k\" (UID: \"c9533b2d-a936-407d-ad6f-e3480bcdd109\") " pod="openshift-image-registry/node-ca-6r78k"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198277    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6bb6b12f-ef56-49af-a24e-9759831b31b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198329    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-var-lib-openvswitch\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198355    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc23afba-cb1c-4664-badc-e76289c18e6d-ovnkube-script-lib\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-device-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-etc-selinux\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198435    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-sys-fs\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b7d0ed0-8337-4b8c-adfa-4918f5150593-hosts-file\") pod \"node-resolver-7lxrw\" (UID: \"7b7d0ed0-8337-4b8c-adfa-4918f5150593\") " pod="openshift-dns/node-resolver-7lxrw"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198518    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-node-log\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-cni-netd\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6bb6b12f-ef56-49af-a24e-9759831b31b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-var-lib-kubelet\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-dbus\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-device-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-etc-openvswitch\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198648    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b7d0ed0-8337-4b8c-adfa-4918f5150593-hosts-file\") pod \"node-resolver-7lxrw\" (UID: \"7b7d0ed0-8337-4b8c-adfa-4918f5150593\") " pod="openshift-dns/node-resolver-7lxrw"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-system-cni-dir\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.200900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198745    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-sys-fs\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198777    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-cni-dir\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198805    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-socket-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198830    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b7d0ed0-8337-4b8c-adfa-4918f5150593-tmp-dir\") pod \"node-resolver-7lxrw\" (UID: \"7b7d0ed0-8337-4b8c-adfa-4918f5150593\") " pod="openshift-dns/node-resolver-7lxrw"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n64r\" (UniqueName: \"kubernetes.io/projected/6bb6b12f-ef56-49af-a24e-9759831b31b7-kube-api-access-5n64r\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-run-systemd\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198906    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-var-lib-cni-multus\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198933    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-tuned\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.198957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721c30dc-143f-4729-883a-12be9fd78201-host-slash\") pod \"iptables-alerter-9qb5m\" (UID: \"721c30dc-143f-4729-883a-12be9fd78201\") " pod="openshift-network-operator/iptables-alerter-9qb5m"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6bb6b12f-ef56-49af-a24e-9759831b31b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199076    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-var-lib-cni-bin\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hszsm\" (UniqueName: \"kubernetes.io/projected/dc23afba-cb1c-4664-badc-e76289c18e6d-kube-api-access-hszsm\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199132    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-var-lib-kubelet\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/84d7fdb2-524e-4627-8740-17ba2dcfefa0-konnectivity-ca\") pod \"konnectivity-agent-rm7gp\" (UID: \"84d7fdb2-524e-4627-8740-17ba2dcfefa0\") " pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199188    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2sfl\" (UniqueName: \"kubernetes.io/projected/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-kube-api-access-t2sfl\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199193    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-var-lib-kubelet\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199212    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-run\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.201719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199239    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/721c30dc-143f-4729-883a-12be9fd78201-iptables-alerter-script\") pod \"iptables-alerter-9qb5m\" (UID: \"721c30dc-143f-4729-883a-12be9fd78201\") " pod="openshift-network-operator/iptables-alerter-9qb5m"
Sep 11 14:39:56.202565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199161    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-dbus\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:56.202565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199343    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-run-ovn\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.202565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199377    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.202565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199409    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc23afba-cb1c-4664-badc-e76289c18e6d-ovnkube-config\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.202565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-hostroot\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.202565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199449    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-socket-dir\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.202565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63d954c0-13c8-4f70-bf69-ce8c999c60ea-run\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.202565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199586    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-etc-selinux\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.202565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.199735    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b7d0ed0-8337-4b8c-adfa-4918f5150593-tmp-dir\") pod \"node-resolver-7lxrw\" (UID: \"7b7d0ed0-8337-4b8c-adfa-4918f5150593\") " pod="openshift-dns/node-resolver-7lxrw"
Sep 11 14:39:56.203614 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.203523    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63d954c0-13c8-4f70-bf69-ce8c999c60ea-tmp\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.203732 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.203642    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63d954c0-13c8-4f70-bf69-ce8c999c60ea-etc-tuned\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.206170 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.206143    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqpg\" (UniqueName: \"kubernetes.io/projected/63d954c0-13c8-4f70-bf69-ce8c999c60ea-kube-api-access-wxqpg\") pod \"tuned-5bk6j\" (UID: \"63d954c0-13c8-4f70-bf69-ce8c999c60ea\") " pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.207421 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.207401    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7zr7\" (UniqueName: \"kubernetes.io/projected/7b7d0ed0-8337-4b8c-adfa-4918f5150593-kube-api-access-t7zr7\") pod \"node-resolver-7lxrw\" (UID: \"7b7d0ed0-8337-4b8c-adfa-4918f5150593\") " pod="openshift-dns/node-resolver-7lxrw"
Sep 11 14:39:56.207807 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.207786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2sfl\" (UniqueName: \"kubernetes.io/projected/614ef36a-52e8-4ae0-ae06-4df0e11b7a56-kube-api-access-t2sfl\") pod \"aws-ebs-csi-driver-node-76ptf\" (UID: \"614ef36a-52e8-4ae0-ae06-4df0e11b7a56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.209077 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.209044    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n64r\" (UniqueName: \"kubernetes.io/projected/6bb6b12f-ef56-49af-a24e-9759831b31b7-kube-api-access-5n64r\") pod \"multus-additional-cni-plugins-8r5kp\" (UID: \"6bb6b12f-ef56-49af-a24e-9759831b31b7\") " pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.300400 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-node-log\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.300400 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300403    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-cni-netd\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-etc-openvswitch\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-system-cni-dir\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-cni-dir\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-run-systemd\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300509    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-node-log\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300509    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-cni-netd\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-var-lib-cni-multus\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300553    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721c30dc-143f-4729-883a-12be9fd78201-host-slash\") pod \"iptables-alerter-9qb5m\" (UID: \"721c30dc-143f-4729-883a-12be9fd78201\") " pod="openshift-network-operator/iptables-alerter-9qb5m"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-var-lib-cni-bin\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hszsm\" (UniqueName: \"kubernetes.io/projected/dc23afba-cb1c-4664-badc-e76289c18e6d-kube-api-access-hszsm\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300605    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-var-lib-cni-multus\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300572    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-etc-openvswitch\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.300633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300622    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-var-lib-kubelet\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-cni-dir\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300681    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/84d7fdb2-524e-4627-8740-17ba2dcfefa0-konnectivity-ca\") pod \"konnectivity-agent-rm7gp\" (UID: \"84d7fdb2-524e-4627-8740-17ba2dcfefa0\") " pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-var-lib-kubelet\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300597    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-run-systemd\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721c30dc-143f-4729-883a-12be9fd78201-host-slash\") pod \"iptables-alerter-9qb5m\" (UID: \"721c30dc-143f-4729-883a-12be9fd78201\") " pod="openshift-network-operator/iptables-alerter-9qb5m"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300744    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-var-lib-cni-bin\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/721c30dc-143f-4729-883a-12be9fd78201-iptables-alerter-script\") pod \"iptables-alerter-9qb5m\" (UID: \"721c30dc-143f-4729-883a-12be9fd78201\") " pod="openshift-network-operator/iptables-alerter-9qb5m"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300769    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-system-cni-dir\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-run-ovn\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300891    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300900    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-run-ovn\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc23afba-cb1c-4664-badc-e76289c18e6d-ovnkube-config\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.300971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-hostroot\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301024    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-slash\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301081    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-hostroot\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301098    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-log-socket\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.301308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301123    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-slash\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-conf-dir\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301147    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-log-socket\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301155    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sv72\" (UniqueName: \"kubernetes.io/projected/a8003a60-3e02-4e75-b168-c540a3975f4d-kube-api-access-9sv72\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301161    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-conf-dir\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301204    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-run-netns\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301229    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9533b2d-a936-407d-ad6f-e3480bcdd109-serviceca\") pod \"node-ca-6r78k\" (UID: \"c9533b2d-a936-407d-ad6f-e3480bcdd109\") " pod="openshift-image-registry/node-ca-6r78k"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-cni-bin\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301276    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/84d7fdb2-524e-4627-8740-17ba2dcfefa0-konnectivity-ca\") pod \"konnectivity-agent-rm7gp\" (UID: \"84d7fdb2-524e-4627-8740-17ba2dcfefa0\") " pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-run-k8s-cni-cncf-io\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-run-k8s-cni-cncf-io\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301311    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/721c30dc-143f-4729-883a-12be9fd78201-iptables-alerter-script\") pod \"iptables-alerter-9qb5m\" (UID: \"721c30dc-143f-4729-883a-12be9fd78201\") " pod="openshift-network-operator/iptables-alerter-9qb5m"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-run-netns\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301335    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-kubelet\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301393    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.301409    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-daemon-config\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301445    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-kubelet\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.301473    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs podName:fde006ea-2cba-4b23-b83e-6f7703193a31 nodeName:}" failed. No retries permitted until 2026-09-11 14:39:56.801437253 +0000 UTC m=+3.124065601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs") pod "network-metrics-daemon-2snh4" (UID: "fde006ea-2cba-4b23-b83e-6f7703193a31") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301485    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc23afba-cb1c-4664-badc-e76289c18e6d-ovnkube-config\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-etc-kubernetes\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301503    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-cni-bin\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301529    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-cnibin\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301553    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-etc-kubernetes\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-socket-dir-parent\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-run-multus-certs\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301600    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-cnibin\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczgc\" (UniqueName: \"kubernetes.io/projected/c9533b2d-a936-407d-ad6f-e3480bcdd109-kube-api-access-fczgc\") pod \"node-ca-6r78k\" (UID: \"c9533b2d-a936-407d-ad6f-e3480bcdd109\") " pod="openshift-image-registry/node-ca-6r78k"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zc5q\" (UniqueName: \"kubernetes.io/projected/fde006ea-2cba-4b23-b83e-6f7703193a31-kube-api-access-8zc5q\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-socket-dir-parent\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvm47\" (UniqueName: \"kubernetes.io/projected/721c30dc-143f-4729-883a-12be9fd78201-kube-api-access-bvm47\") pod \"iptables-alerter-9qb5m\" (UID: \"721c30dc-143f-4729-883a-12be9fd78201\") " pod="openshift-network-operator/iptables-alerter-9qb5m"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301674    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8003a60-3e02-4e75-b168-c540a3975f4d-host-run-multus-certs\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9533b2d-a936-407d-ad6f-e3480bcdd109-serviceca\") pod \"node-ca-6r78k\" (UID: \"c9533b2d-a936-407d-ad6f-e3480bcdd109\") " pod="openshift-image-registry/node-ca-6r78k"
Sep 11 14:39:56.302933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-run-openvswitch\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301721    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/84d7fdb2-524e-4627-8740-17ba2dcfefa0-agent-certs\") pod \"konnectivity-agent-rm7gp\" (UID: \"84d7fdb2-524e-4627-8740-17ba2dcfefa0\") " pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301755    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-run-netns\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301722    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-run-openvswitch\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc23afba-cb1c-4664-badc-e76289c18e6d-env-overrides\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-host-run-netns\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301840    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc23afba-cb1c-4664-badc-e76289c18e6d-ovn-node-metrics-cert\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301870    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nm9\" (UniqueName: \"kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9\") pod \"network-check-target-9l6tg\" (UID: \"4bd24af6-f410-4833-816b-772fa4da84f1\") " pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8003a60-3e02-4e75-b168-c540a3975f4d-multus-daemon-config\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.301974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-systemd-units\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.302006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8003a60-3e02-4e75-b168-c540a3975f4d-cni-binary-copy\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.302031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9533b2d-a936-407d-ad6f-e3480bcdd109-host\") pod \"node-ca-6r78k\" (UID: \"c9533b2d-a936-407d-ad6f-e3480bcdd109\") " pod="openshift-image-registry/node-ca-6r78k"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.302037    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-systemd-units\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.302058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-var-lib-openvswitch\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.302088    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc23afba-cb1c-4664-badc-e76289c18e6d-ovnkube-script-lib\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.302099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9533b2d-a936-407d-ad6f-e3480bcdd109-host\") pod \"node-ca-6r78k\" (UID: \"c9533b2d-a936-407d-ad6f-e3480bcdd109\") " pod="openshift-image-registry/node-ca-6r78k"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.302145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc23afba-cb1c-4664-badc-e76289c18e6d-env-overrides\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.302145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc23afba-cb1c-4664-badc-e76289c18e6d-var-lib-openvswitch\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.303493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.302470    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8003a60-3e02-4e75-b168-c540a3975f4d-cni-binary-copy\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.304196 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.302614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc23afba-cb1c-4664-badc-e76289c18e6d-ovnkube-script-lib\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.304394 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.304365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc23afba-cb1c-4664-badc-e76289c18e6d-ovn-node-metrics-cert\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.304672 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.304555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/84d7fdb2-524e-4627-8740-17ba2dcfefa0-agent-certs\") pod \"konnectivity-agent-rm7gp\" (UID: \"84d7fdb2-524e-4627-8740-17ba2dcfefa0\") " pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:39:56.311431 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.311405    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczgc\" (UniqueName: \"kubernetes.io/projected/c9533b2d-a936-407d-ad6f-e3480bcdd109-kube-api-access-fczgc\") pod \"node-ca-6r78k\" (UID: \"c9533b2d-a936-407d-ad6f-e3480bcdd109\") " pod="openshift-image-registry/node-ca-6r78k"
Sep 11 14:39:56.312137 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.312113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hszsm\" (UniqueName: \"kubernetes.io/projected/dc23afba-cb1c-4664-badc-e76289c18e6d-kube-api-access-hszsm\") pod \"ovnkube-node-d6xtk\" (UID: \"dc23afba-cb1c-4664-badc-e76289c18e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.312284 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.312226    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:39:56.312284 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.312249    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:39:56.312284 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.312263    2559 projected.go:196] Error preparing data for projected volume kube-api-access-89nm9 for pod openshift-network-diagnostics/network-check-target-9l6tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:39:56.312448 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.312321    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9 podName:4bd24af6-f410-4833-816b-772fa4da84f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:39:56.812305159 +0000 UTC m=+3.134933507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-89nm9" (UniqueName: "kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9") pod "network-check-target-9l6tg" (UID: "4bd24af6-f410-4833-816b-772fa4da84f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:39:56.313703 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.313679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zc5q\" (UniqueName: \"kubernetes.io/projected/fde006ea-2cba-4b23-b83e-6f7703193a31-kube-api-access-8zc5q\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:39:56.313967 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.313942    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvm47\" (UniqueName: \"kubernetes.io/projected/721c30dc-143f-4729-883a-12be9fd78201-kube-api-access-bvm47\") pod \"iptables-alerter-9qb5m\" (UID: \"721c30dc-143f-4729-883a-12be9fd78201\") " pod="openshift-network-operator/iptables-alerter-9qb5m"
Sep 11 14:39:56.314050 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.313948    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sv72\" (UniqueName: \"kubernetes.io/projected/a8003a60-3e02-4e75-b168-c540a3975f4d-kube-api-access-9sv72\") pod \"multus-2gmn2\" (UID: \"a8003a60-3e02-4e75-b168-c540a3975f4d\") " pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.392379 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.392331    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8r5kp"
Sep 11 14:39:56.402070 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.402042    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf"
Sep 11 14:39:56.412071 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.412045    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5bk6j"
Sep 11 14:39:56.418710 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.418688    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7lxrw"
Sep 11 14:39:56.425359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.425339    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6r78k"
Sep 11 14:39:56.434026 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.434007    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9qb5m"
Sep 11 14:39:56.441590 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.441571    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:39:56.449068 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.449052    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2gmn2"
Sep 11 14:39:56.454466 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.454438    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:39:56.705783 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.705703    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:56.705939 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.705830    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:39:56.705939 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.705901    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret podName:c89e9a20-c4e4-4c03-9601-8e47f2ad2df2 nodeName:}" failed. No retries permitted until 2026-09-11 14:39:57.705883624 +0000 UTC m=+4.028511995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret") pod "global-pull-secret-syncer-24976" (UID: "c89e9a20-c4e4-4c03-9601-8e47f2ad2df2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:39:56.806679 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.806640    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:39:56.806830 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.806799    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:39:56.806881 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.806868    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs podName:fde006ea-2cba-4b23-b83e-6f7703193a31 nodeName:}" failed. No retries permitted until 2026-09-11 14:39:57.806851582 +0000 UTC m=+4.129479934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs") pod "network-metrics-daemon-2snh4" (UID: "fde006ea-2cba-4b23-b83e-6f7703193a31") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:39:56.907269 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:56.907230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nm9\" (UniqueName: \"kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9\") pod \"network-check-target-9l6tg\" (UID: \"4bd24af6-f410-4833-816b-772fa4da84f1\") " pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:39:56.907438 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.907406    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:39:56.907438 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.907433    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:39:56.907548 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.907448    2559 projected.go:196] Error preparing data for projected volume kube-api-access-89nm9 for pod openshift-network-diagnostics/network-check-target-9l6tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:39:56.907548 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:56.907524    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9 podName:4bd24af6-f410-4833-816b-772fa4da84f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:39:57.907509761 +0000 UTC m=+4.230138108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-89nm9" (UniqueName: "kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9") pod "network-check-target-9l6tg" (UID: "4bd24af6-f410-4833-816b-772fa4da84f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:39:57.152280 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:57.152213    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:34:56 +0000 UTC" deadline="2028-02-06 13:22:40.739556396 +0000 UTC"
Sep 11 14:39:57.152280 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:57.152273    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12310h42m43.587288206s"
Sep 11 14:39:57.309138 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:57.309112    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d7fdb2_524e_4627_8740_17ba2dcfefa0.slice/crio-10be0a5555495cf2e8be46fdad12cab8258c2e4dd383b4477299564850d877dc WatchSource:0}: Error finding container 10be0a5555495cf2e8be46fdad12cab8258c2e4dd383b4477299564850d877dc: Status 404 returned error can't find the container with id 10be0a5555495cf2e8be46fdad12cab8258c2e4dd383b4477299564850d877dc
Sep 11 14:39:57.310696 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:57.310570    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721c30dc_143f_4729_883a_12be9fd78201.slice/crio-0f9c16756146a2991edf31dc3bce97d687c390a4e56c03ba2213fa1a03836573 WatchSource:0}: Error finding container 0f9c16756146a2991edf31dc3bce97d687c390a4e56c03ba2213fa1a03836573: Status 404 returned error can't find the container with id 0f9c16756146a2991edf31dc3bce97d687c390a4e56c03ba2213fa1a03836573
Sep 11 14:39:57.314037 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:57.313946    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc23afba_cb1c_4664_badc_e76289c18e6d.slice/crio-a3812b316282811de65f03fb1d9d9eab26ebda3db9442264ae33724cabc410e2 WatchSource:0}: Error finding container a3812b316282811de65f03fb1d9d9eab26ebda3db9442264ae33724cabc410e2: Status 404 returned error can't find the container with id a3812b316282811de65f03fb1d9d9eab26ebda3db9442264ae33724cabc410e2
Sep 11 14:39:57.316225 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:57.316121    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb6b12f_ef56_49af_a24e_9759831b31b7.slice/crio-acf719387859a3997c37a1dab7407ace7b732b2b5c437b3931b4210651ec0700 WatchSource:0}: Error finding container acf719387859a3997c37a1dab7407ace7b732b2b5c437b3931b4210651ec0700: Status 404 returned error can't find the container with id acf719387859a3997c37a1dab7407ace7b732b2b5c437b3931b4210651ec0700
Sep 11 14:39:57.318590 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:57.317706    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7d0ed0_8337_4b8c_adfa_4918f5150593.slice/crio-26199fc05fab24f05bbb63927ac2c8b80f633fa7f7c0b601c8351f32fdc012c1 WatchSource:0}: Error finding container 26199fc05fab24f05bbb63927ac2c8b80f633fa7f7c0b601c8351f32fdc012c1: Status 404 returned error can't find the container with id 26199fc05fab24f05bbb63927ac2c8b80f633fa7f7c0b601c8351f32fdc012c1
Sep 11 14:39:57.318590 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:57.318080    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9533b2d_a936_407d_ad6f_e3480bcdd109.slice/crio-c0935eeae69996281796d0147db03b25e7b08df0fc6c329ac420bd8cde13bd53 WatchSource:0}: Error finding container c0935eeae69996281796d0147db03b25e7b08df0fc6c329ac420bd8cde13bd53: Status 404 returned error can't find the container with id c0935eeae69996281796d0147db03b25e7b08df0fc6c329ac420bd8cde13bd53
Sep 11 14:39:57.319392 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:57.319343    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d954c0_13c8_4f70_bf69_ce8c999c60ea.slice/crio-29c9975b54b86acc6f453b77083805070ced9f57abb2962d41a609c376d9aeca WatchSource:0}: Error finding container 29c9975b54b86acc6f453b77083805070ced9f57abb2962d41a609c376d9aeca: Status 404 returned error can't find the container with id 29c9975b54b86acc6f453b77083805070ced9f57abb2962d41a609c376d9aeca
Sep 11 14:39:57.321834 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:39:57.321014    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8003a60_3e02_4e75_b168_c540a3975f4d.slice/crio-6b7f646bee17e9a7e66678def31b3ec9da00ed2b83cbeb1f5a0dff56d51378ee WatchSource:0}: Error finding container 6b7f646bee17e9a7e66678def31b3ec9da00ed2b83cbeb1f5a0dff56d51378ee: Status 404 returned error can't find the container with id 6b7f646bee17e9a7e66678def31b3ec9da00ed2b83cbeb1f5a0dff56d51378ee
Sep 11 14:39:57.713965 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:57.713924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:57.714138 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:57.714088    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:39:57.714181 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:57.714164    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret podName:c89e9a20-c4e4-4c03-9601-8e47f2ad2df2 nodeName:}" failed. No retries permitted until 2026-09-11 14:39:59.714147971 +0000 UTC m=+6.036776322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret") pod "global-pull-secret-syncer-24976" (UID: "c89e9a20-c4e4-4c03-9601-8e47f2ad2df2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:39:57.815373 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:57.815275    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:39:57.815555 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:57.815464    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:39:57.815555 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:57.815528    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs podName:fde006ea-2cba-4b23-b83e-6f7703193a31 nodeName:}" failed. No retries permitted until 2026-09-11 14:39:59.815510111 +0000 UTC m=+6.138138458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs") pod "network-metrics-daemon-2snh4" (UID: "fde006ea-2cba-4b23-b83e-6f7703193a31") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:39:57.916656 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:57.916437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nm9\" (UniqueName: \"kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9\") pod \"network-check-target-9l6tg\" (UID: \"4bd24af6-f410-4833-816b-772fa4da84f1\") " pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:39:57.916881 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:57.916601    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:39:57.916881 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:57.916749    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:39:57.916881 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:57.916763    2559 projected.go:196] Error preparing data for projected volume kube-api-access-89nm9 for pod openshift-network-diagnostics/network-check-target-9l6tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:39:57.916881 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:57.916822    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9 podName:4bd24af6-f410-4833-816b-772fa4da84f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:39:59.916804585 +0000 UTC m=+6.239432948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-89nm9" (UniqueName: "kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9") pod "network-check-target-9l6tg" (UID: "4bd24af6-f410-4833-816b-772fa4da84f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:39:58.153420 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.153309    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:34:56 +0000 UTC" deadline="2028-05-10 06:23:41.377446275 +0000 UTC"
Sep 11 14:39:58.153420 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.153345    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14559h43m43.224105827s"
Sep 11 14:39:58.175980 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.174572    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:58.175980 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:58.174704    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:39:58.175980 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.175093    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:39:58.175980 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:58.175196    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:39:58.175980 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.175255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:39:58.175980 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:58.175350    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:39:58.194123 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.194088    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5bk6j" event={"ID":"63d954c0-13c8-4f70-bf69-ce8c999c60ea","Type":"ContainerStarted","Data":"29c9975b54b86acc6f453b77083805070ced9f57abb2962d41a609c376d9aeca"}
Sep 11 14:39:58.204516 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.204415    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9qb5m" event={"ID":"721c30dc-143f-4729-883a-12be9fd78201","Type":"ContainerStarted","Data":"0f9c16756146a2991edf31dc3bce97d687c390a4e56c03ba2213fa1a03836573"}
Sep 11 14:39:58.210742 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.210713    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rm7gp" event={"ID":"84d7fdb2-524e-4627-8740-17ba2dcfefa0","Type":"ContainerStarted","Data":"10be0a5555495cf2e8be46fdad12cab8258c2e4dd383b4477299564850d877dc"}
Sep 11 14:39:58.229757 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.229726    2559 generic.go:356] "Generic (PLEG): container finished" podID="73cb3b572770e472576c1c3acbe52a84" containerID="86e3c2089547436efb1c8d0b692a4d042b75e8d1ca4d7adfb0716532beafdb0b" exitCode=0
Sep 11 14:39:58.229891 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.229801    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal" event={"ID":"73cb3b572770e472576c1c3acbe52a84","Type":"ContainerDied","Data":"86e3c2089547436efb1c8d0b692a4d042b75e8d1ca4d7adfb0716532beafdb0b"}
Sep 11 14:39:58.242560 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.242534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-223.ec2.internal" event={"ID":"b26ed7e8c79abb7d6bd735b73ba0b5d9","Type":"ContainerStarted","Data":"e556147596496fbba7a6eed822d6fb16927f5ea60c46ad066a5ddf7195a01abc"}
Sep 11 14:39:58.259002 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.258945    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2gmn2" event={"ID":"a8003a60-3e02-4e75-b168-c540a3975f4d","Type":"ContainerStarted","Data":"6b7f646bee17e9a7e66678def31b3ec9da00ed2b83cbeb1f5a0dff56d51378ee"}
Sep 11 14:39:58.262753 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.262727    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6r78k" event={"ID":"c9533b2d-a936-407d-ad6f-e3480bcdd109","Type":"ContainerStarted","Data":"c0935eeae69996281796d0147db03b25e7b08df0fc6c329ac420bd8cde13bd53"}
Sep 11 14:39:58.267718 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.267693    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7lxrw" event={"ID":"7b7d0ed0-8337-4b8c-adfa-4918f5150593","Type":"ContainerStarted","Data":"26199fc05fab24f05bbb63927ac2c8b80f633fa7f7c0b601c8351f32fdc012c1"}
Sep 11 14:39:58.271120 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.271069    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-223.ec2.internal" podStartSLOduration=3.271051688 podStartE2EDuration="3.271051688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:39:58.270528751 +0000 UTC m=+4.593157124" watchObservedRunningTime="2026-09-11 14:39:58.271051688 +0000 UTC m=+4.593680062"
Sep 11 14:39:58.294898 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.293799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r5kp" event={"ID":"6bb6b12f-ef56-49af-a24e-9759831b31b7","Type":"ContainerStarted","Data":"acf719387859a3997c37a1dab7407ace7b732b2b5c437b3931b4210651ec0700"}
Sep 11 14:39:58.312531 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.312486    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk" event={"ID":"dc23afba-cb1c-4664-badc-e76289c18e6d","Type":"ContainerStarted","Data":"a3812b316282811de65f03fb1d9d9eab26ebda3db9442264ae33724cabc410e2"}
Sep 11 14:39:58.317869 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:58.317843    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf" event={"ID":"614ef36a-52e8-4ae0-ae06-4df0e11b7a56","Type":"ContainerStarted","Data":"eaa9d5243843bb0ee56426e2aa5319b00d0c5b4b98f7a21488b493e298e0e160"}
Sep 11 14:39:59.336579 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:59.336266    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal" event={"ID":"73cb3b572770e472576c1c3acbe52a84","Type":"ContainerStarted","Data":"997e824db22196531fcc9e033fcb85ba3a7e356c66bf96c119c5acbcda89df32"}
Sep 11 14:39:59.734776 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:59.734734    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:39:59.734956 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:59.734939    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:39:59.735015 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:59.735002    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret podName:c89e9a20-c4e4-4c03-9601-8e47f2ad2df2 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:03.73498267 +0000 UTC m=+10.057611034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret") pod "global-pull-secret-syncer-24976" (UID: "c89e9a20-c4e4-4c03-9601-8e47f2ad2df2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:39:59.836226 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:59.836184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:39:59.836427 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:59.836381    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:39:59.836519 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:59.836446    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs podName:fde006ea-2cba-4b23-b83e-6f7703193a31 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:03.836429114 +0000 UTC m=+10.159057479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs") pod "network-metrics-daemon-2snh4" (UID: "fde006ea-2cba-4b23-b83e-6f7703193a31") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:39:59.937883 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:39:59.937224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nm9\" (UniqueName: \"kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9\") pod \"network-check-target-9l6tg\" (UID: \"4bd24af6-f410-4833-816b-772fa4da84f1\") " pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:39:59.937883 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:59.937464    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:39:59.937883 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:59.937487    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:39:59.937883 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:59.937500    2559 projected.go:196] Error preparing data for projected volume kube-api-access-89nm9 for pod openshift-network-diagnostics/network-check-target-9l6tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:39:59.937883 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:39:59.937558    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9 podName:4bd24af6-f410-4833-816b-772fa4da84f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:03.937540381 +0000 UTC m=+10.260168744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-89nm9" (UniqueName: "kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9") pod "network-check-target-9l6tg" (UID: "4bd24af6-f410-4833-816b-772fa4da84f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:00.176962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:00.176346    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:00.176962 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:00.176499    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:00.176962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:00.176606    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:00.176962 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:00.176724    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:00.176962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:00.176783    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:00.176962 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:00.176848    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:02.174038 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:02.173724    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:02.174038 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:02.173731    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:02.174038 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:02.173870    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:02.174038 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:02.173940    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:02.174038 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:02.173990    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:02.174741 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:02.174058    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:03.770094 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:03.770051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:03.770580 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:03.770162    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:03.770580 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:03.770223    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret podName:c89e9a20-c4e4-4c03-9601-8e47f2ad2df2 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:11.770206684 +0000 UTC m=+18.092835032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret") pod "global-pull-secret-syncer-24976" (UID: "c89e9a20-c4e4-4c03-9601-8e47f2ad2df2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:03.870747 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:03.870710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:03.870747 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:03.870827    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:03.871138 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:03.870885    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs podName:fde006ea-2cba-4b23-b83e-6f7703193a31 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:11.870866361 +0000 UTC m=+18.193494711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs") pod "network-metrics-daemon-2snh4" (UID: "fde006ea-2cba-4b23-b83e-6f7703193a31") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:03.972064 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:03.971998    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nm9\" (UniqueName: \"kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9\") pod \"network-check-target-9l6tg\" (UID: \"4bd24af6-f410-4833-816b-772fa4da84f1\") " pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:03.972228 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:03.972166    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:40:03.972228 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:03.972191    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:40:03.972228 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:03.972205    2559 projected.go:196] Error preparing data for projected volume kube-api-access-89nm9 for pod openshift-network-diagnostics/network-check-target-9l6tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:03.972367 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:03.972266    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9 podName:4bd24af6-f410-4833-816b-772fa4da84f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:11.972247467 +0000 UTC m=+18.294875840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-89nm9" (UniqueName: "kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9") pod "network-check-target-9l6tg" (UID: "4bd24af6-f410-4833-816b-772fa4da84f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:04.174608 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:04.174470    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:04.174608 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:04.174586    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:04.175150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:04.174891    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:04.175150 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:04.174993    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:04.175150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:04.175044    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:04.175150 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:04.175103    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:06.173688 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:06.173658    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:06.174045 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:06.173707    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:06.174045 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:06.173809    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:06.174145 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:06.174113    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:06.174198 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:06.174180    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:06.174297 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:06.174263    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:08.174301 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:08.174266    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:08.174733 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:08.174273    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:08.174733 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:08.174414    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:08.174733 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:08.174266    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:08.174733 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:08.174493    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:08.174733 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:08.174559    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:10.173909 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:10.173879    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:10.173909 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:10.173910    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:10.174397 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:10.173879    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:10.174397 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:10.174005    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:10.174397 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:10.174095    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:10.174397 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:10.174179    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:11.831935 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:11.831883    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:11.832397 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:11.832049    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:11.832397 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:11.832139    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret podName:c89e9a20-c4e4-4c03-9601-8e47f2ad2df2 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:27.832122972 +0000 UTC m=+34.154751320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret") pod "global-pull-secret-syncer-24976" (UID: "c89e9a20-c4e4-4c03-9601-8e47f2ad2df2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:11.933190 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:11.933152    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:11.933364 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:11.933339    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:11.933435 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:11.933424    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs podName:fde006ea-2cba-4b23-b83e-6f7703193a31 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:27.933402774 +0000 UTC m=+34.256031139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs") pod "network-metrics-daemon-2snh4" (UID: "fde006ea-2cba-4b23-b83e-6f7703193a31") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:12.033873 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:12.033831    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nm9\" (UniqueName: \"kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9\") pod \"network-check-target-9l6tg\" (UID: \"4bd24af6-f410-4833-816b-772fa4da84f1\") " pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:12.034053 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:12.033983    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:40:12.034053 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:12.034011    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:40:12.034053 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:12.034023    2559 projected.go:196] Error preparing data for projected volume kube-api-access-89nm9 for pod openshift-network-diagnostics/network-check-target-9l6tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:12.034185 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:12.034079    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9 podName:4bd24af6-f410-4833-816b-772fa4da84f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:28.034063182 +0000 UTC m=+34.356691530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-89nm9" (UniqueName: "kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9") pod "network-check-target-9l6tg" (UID: "4bd24af6-f410-4833-816b-772fa4da84f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:12.174097 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:12.174009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:12.174253 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:12.174024    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:12.174253 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:12.174165    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:12.174253 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:12.174229    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:12.174446 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:12.174266    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:12.174446 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:12.174360    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:14.175301 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:14.175254    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:14.175771 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:14.175377    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:14.175771 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:14.175380    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:14.175771 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:14.175444    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:14.175771 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:14.175496    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:14.175771 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:14.175562    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:15.367678 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:15.367598    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rm7gp" event={"ID":"84d7fdb2-524e-4627-8740-17ba2dcfefa0","Type":"ContainerStarted","Data":"bf1b93775336d540b8ae55d5b8226569a70a2a76f76c4e20f7762bc9c5680244"}
Sep 11 14:40:15.370336 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:15.370166    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6r78k" event={"ID":"c9533b2d-a936-407d-ad6f-e3480bcdd109","Type":"ContainerStarted","Data":"619b604b78b1ca6fcd7972bd2539ae33b5e812410893dab41d207cea2285e492"}
Sep 11 14:40:15.371696 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:15.371670    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7lxrw" event={"ID":"7b7d0ed0-8337-4b8c-adfa-4918f5150593","Type":"ContainerStarted","Data":"6ce070cb83c52c9259cac4db21aeba0d5e395a6ea140c720ab2da1921b873aa2"}
Sep 11 14:40:15.374669 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:15.374644    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r5kp" event={"ID":"6bb6b12f-ef56-49af-a24e-9759831b31b7","Type":"ContainerStarted","Data":"ea2a4092bce7b705424e02cfbb2fbbd88d9bea750c610e19009cd8d9c734c61f"}
Sep 11 14:40:15.376837 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:15.376733    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk" event={"ID":"dc23afba-cb1c-4664-badc-e76289c18e6d","Type":"ContainerStarted","Data":"89a8b165defebb13dfd29f4ea2584b77194b267db69272df2858123081f89112"}
Sep 11 14:40:15.378853 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:15.378795    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf" event={"ID":"614ef36a-52e8-4ae0-ae06-4df0e11b7a56","Type":"ContainerStarted","Data":"aad1ec407f71f9429641b159c697f27b5cf3d586982dd3f01ea0c15d39127b86"}
Sep 11 14:40:15.381139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:15.380555    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5bk6j" event={"ID":"63d954c0-13c8-4f70-bf69-ce8c999c60ea","Type":"ContainerStarted","Data":"c0778fa5a88a41123ae55e3e61a932a4e1e78377437f4a734329f244575d8b0b"}
Sep 11 14:40:15.393181 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:15.393133    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-223.ec2.internal" podStartSLOduration=20.393118347 podStartE2EDuration="20.393118347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:39:59.358817125 +0000 UTC m=+5.681445496" watchObservedRunningTime="2026-09-11 14:40:15.393118347 +0000 UTC m=+21.715746719"
Sep 11 14:40:15.393709 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:15.393661    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rm7gp" podStartSLOduration=3.856530843 podStartE2EDuration="21.393645492s" totalImagesPullingTime="17.537114649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:15.393477247 +0000 UTC m=+21.716105615" watchObservedRunningTime="2026-09-11 14:40:15.393645492 +0000 UTC m=+21.716273864"
Sep 11 14:40:15.439300 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:15.439211    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7lxrw" podStartSLOduration=3.63603945 podStartE2EDuration="21.439188546s" totalImagesPullingTime="17.803149096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:15.438726177 +0000 UTC m=+21.761354546" watchObservedRunningTime="2026-09-11 14:40:15.439188546 +0000 UTC m=+21.761816918"
Sep 11 14:40:15.456193 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:15.456127    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6r78k" podStartSLOduration=3.67502659 podStartE2EDuration="21.456109953s" totalImagesPullingTime="17.781083363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:15.455779453 +0000 UTC m=+21.778407824" watchObservedRunningTime="2026-09-11 14:40:15.456109953 +0000 UTC m=+21.778738322"
Sep 11 14:40:16.174327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.174065    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:16.174526 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.174067    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:16.174526 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:16.174449    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:16.174526 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.174111    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:16.174710 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:16.174532    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:16.174710 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:16.174614    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:16.384760 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.384722    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2gmn2" event={"ID":"a8003a60-3e02-4e75-b168-c540a3975f4d","Type":"ContainerStarted","Data":"1d499ae909f88f6a3a4693988b0e479a9306935306286663eefef1dfdf8ef1b2"}
Sep 11 14:40:16.386240 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.386214    2559 generic.go:356] "Generic (PLEG): container finished" podID="6bb6b12f-ef56-49af-a24e-9759831b31b7" containerID="ea2a4092bce7b705424e02cfbb2fbbd88d9bea750c610e19009cd8d9c734c61f" exitCode=0
Sep 11 14:40:16.386355 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.386287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r5kp" event={"ID":"6bb6b12f-ef56-49af-a24e-9759831b31b7","Type":"ContainerDied","Data":"ea2a4092bce7b705424e02cfbb2fbbd88d9bea750c610e19009cd8d9c734c61f"}
Sep 11 14:40:16.389989 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.389968    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk" event={"ID":"dc23afba-cb1c-4664-badc-e76289c18e6d","Type":"ContainerStarted","Data":"d32677c0f08af76a1d242623a626484a96779501b1ecd16485b309a60185081c"}
Sep 11 14:40:16.390099 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.390046    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk" event={"ID":"dc23afba-cb1c-4664-badc-e76289c18e6d","Type":"ContainerStarted","Data":"ad9d695efe9d3c3ce04becdf621d86f5ac29c77557f8663be14647d557f67cef"}
Sep 11 14:40:16.390165 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.390097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk" event={"ID":"dc23afba-cb1c-4664-badc-e76289c18e6d","Type":"ContainerStarted","Data":"8a8df04c0efac8488d71c4771a4704455ae36c7dc701986e1c8609d0f6ce96d8"}
Sep 11 14:40:16.390165 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.390150    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk" event={"ID":"dc23afba-cb1c-4664-badc-e76289c18e6d","Type":"ContainerStarted","Data":"711026a841a1f4950c49a7ac1fc254b442d4c6cc9f61f6cde7410bb26414b482"}
Sep 11 14:40:16.390265 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.390199    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk" event={"ID":"dc23afba-cb1c-4664-badc-e76289c18e6d","Type":"ContainerStarted","Data":"bf1209e26ba56a4a16f66e2fb4d23f19d1ec4c294e4b2a3160a23ca49d81c264"}
Sep 11 14:40:16.403664 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.403619    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2gmn2" podStartSLOduration=4.372937844 podStartE2EDuration="22.403605543s" totalImagesPullingTime="18.030667699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:16.403597143 +0000 UTC m=+22.726225514" watchObservedRunningTime="2026-09-11 14:40:16.403605543 +0000 UTC m=+22.726233912"
Sep 11 14:40:16.403832 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:16.403811    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5bk6j" podStartSLOduration=4.603318007 podStartE2EDuration="22.403806136s" totalImagesPullingTime="17.800488129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:15.485687598 +0000 UTC m=+21.808315969" watchObservedRunningTime="2026-09-11 14:40:16.403806136 +0000 UTC m=+22.726434507"
Sep 11 14:40:17.069286 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:17.069254    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 14:40:17.150957 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:17.150790    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T14:40:17.069279675Z","UUID":"35a016f8-f6c8-42cc-9bd1-a6b9954e7676","Handler":null,"Name":"","Endpoint":""}
Sep 11 14:40:17.152677 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:17.152648    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 14:40:17.152814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:17.152687    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 14:40:17.394273 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:17.394233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf" event={"ID":"614ef36a-52e8-4ae0-ae06-4df0e11b7a56","Type":"ContainerStarted","Data":"9f8a87a87c402b49dc5f3e10fcf1e639130b2a06e891b86e88334667befbbaa2"}
Sep 11 14:40:17.395805 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:17.395761    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9qb5m" event={"ID":"721c30dc-143f-4729-883a-12be9fd78201","Type":"ContainerStarted","Data":"64b4d85a3fddd901b0eff628e22ad9a21763ea4e7300d6384ba161393a120aa0"}
Sep 11 14:40:17.412068 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:17.412007    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9qb5m" podStartSLOduration=5.6008638600000005 podStartE2EDuration="23.411984474s" totalImagesPullingTime="17.811120614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:17.411667826 +0000 UTC m=+23.734296197" watchObservedRunningTime="2026-09-11 14:40:17.411984474 +0000 UTC m=+23.734612847"
Sep 11 14:40:18.173752 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:18.173674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:18.173752 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:18.173723    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:18.173967 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:18.173674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:18.173967 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:18.173819    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:18.173967 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:18.173937    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:18.174089 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:18.174026    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:18.402201 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:18.402160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk" event={"ID":"dc23afba-cb1c-4664-badc-e76289c18e6d","Type":"ContainerStarted","Data":"cfc685e81712af843915ceacafe2726e41da3f6b869d8ded8ddb0aad776ec8a1"}
Sep 11 14:40:18.404628 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:18.404600    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf" event={"ID":"614ef36a-52e8-4ae0-ae06-4df0e11b7a56","Type":"ContainerStarted","Data":"7ef40a967a89bdc8eaa283b417cdd593766f8821df87db31ae29d90304dc79c5"}
Sep 11 14:40:18.425707 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:18.425611    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76ptf" podStartSLOduration=3.951669727 podStartE2EDuration="24.425594813s" totalImagesPullingTime="20.473925086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:18.425073852 +0000 UTC m=+24.747702223" watchObservedRunningTime="2026-09-11 14:40:18.425594813 +0000 UTC m=+24.748223182"
Sep 11 14:40:19.620501 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:19.620462    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:40:19.621180 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:19.621161    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:40:20.178048 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:20.178015    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:20.178234 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:20.178133    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:20.178234 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:20.178138    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:20.178234 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:20.178164    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:20.178378 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:20.178255    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:20.178378 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:20.178336    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:20.409785 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:20.409529    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:40:20.410165 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:20.410142    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rm7gp"
Sep 11 14:40:22.173742 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:22.173707    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:22.173742 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:22.173727    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:22.174482 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:22.173712    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:22.174482 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:22.173814    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:22.174482 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:22.173912    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:22.174482 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:22.173998    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:22.415862 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:22.415832    2559 generic.go:356] "Generic (PLEG): container finished" podID="6bb6b12f-ef56-49af-a24e-9759831b31b7" containerID="297fee5ed1a1dac5cb13ad1dc8020fe1a444ded18f0c60a1fb6619f9fde00ba3" exitCode=0
Sep 11 14:40:22.416042 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:22.415915    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r5kp" event={"ID":"6bb6b12f-ef56-49af-a24e-9759831b31b7","Type":"ContainerDied","Data":"297fee5ed1a1dac5cb13ad1dc8020fe1a444ded18f0c60a1fb6619f9fde00ba3"}
Sep 11 14:40:22.419736 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:22.419714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk" event={"ID":"dc23afba-cb1c-4664-badc-e76289c18e6d","Type":"ContainerStarted","Data":"7101af7d9a342fcf74cf590bd252fcbfd2b5e08730c208f54f690a1f6d0090d4"}
Sep 11 14:40:23.423531 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.423279    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:40:23.424021 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.423593    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:40:23.424021 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.423644    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:40:23.439726 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.439699    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:40:23.440395 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.440371    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:40:23.449301 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.449254    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk" podStartSLOduration=11.61953322 podStartE2EDuration="29.449237024s" totalImagesPullingTime="17.829703804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:22.475894608 +0000 UTC m=+28.798522978" watchObservedRunningTime="2026-09-11 14:40:23.449237024 +0000 UTC m=+29.771865401"
Sep 11 14:40:23.449564 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.449545    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2snh4"]
Sep 11 14:40:23.449667 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.449640    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:23.449762 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:23.449740    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:23.452431 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.452402    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-24976"]
Sep 11 14:40:23.452578 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.452518    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:23.452641 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:23.452610    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:23.457608 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.457579    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9l6tg"]
Sep 11 14:40:23.457742 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:23.457729    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:23.458401 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:23.457853    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:24.427302 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:24.427270    2559 generic.go:356] "Generic (PLEG): container finished" podID="6bb6b12f-ef56-49af-a24e-9759831b31b7" containerID="dbdee769ec7f6174ecbb27341415c015aa483614be21280dc885807abed3b321" exitCode=0
Sep 11 14:40:24.427703 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:24.427359    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r5kp" event={"ID":"6bb6b12f-ef56-49af-a24e-9759831b31b7","Type":"ContainerDied","Data":"dbdee769ec7f6174ecbb27341415c015aa483614be21280dc885807abed3b321"}
Sep 11 14:40:25.173626 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:25.173538    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:25.173786 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:25.173668    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:25.173786 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:25.173725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:25.173882 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:25.173830    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:25.173882 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:25.173866    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:25.173946 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:25.173930    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:26.434364 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:26.434280    2559 generic.go:356] "Generic (PLEG): container finished" podID="6bb6b12f-ef56-49af-a24e-9759831b31b7" containerID="aaa6e3e2a2ef39a41f1671466402d2d7ebfa92f7fae35c6671465bad6add06b7" exitCode=0
Sep 11 14:40:26.434364 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:26.434333    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r5kp" event={"ID":"6bb6b12f-ef56-49af-a24e-9759831b31b7","Type":"ContainerDied","Data":"aaa6e3e2a2ef39a41f1671466402d2d7ebfa92f7fae35c6671465bad6add06b7"}
Sep 11 14:40:27.174594 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.174556    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:27.174882 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.174564    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:27.174882 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:27.174675    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2snh4" podUID="fde006ea-2cba-4b23-b83e-6f7703193a31"
Sep 11 14:40:27.174882 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.174702    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:27.174882 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:27.174793    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l6tg" podUID="4bd24af6-f410-4833-816b-772fa4da84f1"
Sep 11 14:40:27.174882 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:27.174877    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24976" podUID="c89e9a20-c4e4-4c03-9601-8e47f2ad2df2"
Sep 11 14:40:27.849267 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.849223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:27.849767 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:27.849366    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:27.849767 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:27.849442    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret podName:c89e9a20-c4e4-4c03-9601-8e47f2ad2df2 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:59.849421501 +0000 UTC m=+66.172049865 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret") pod "global-pull-secret-syncer-24976" (UID: "c89e9a20-c4e4-4c03-9601-8e47f2ad2df2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:27.941282 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.941249    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-223.ec2.internal" event="NodeReady"
Sep 11 14:40:27.941421 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.941414    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 14:40:27.950374 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.950346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:27.950546 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:27.950526    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:27.950611 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:27.950603    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs podName:fde006ea-2cba-4b23-b83e-6f7703193a31 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:59.950583156 +0000 UTC m=+66.273211518 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs") pod "network-metrics-daemon-2snh4" (UID: "fde006ea-2cba-4b23-b83e-6f7703193a31") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:27.981403 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.981369    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b7f94f844-q7kdx"]
Sep 11 14:40:27.984050 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.984025    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:27.986610 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.986576    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 14:40:27.986738 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.986586    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 14:40:27.986799 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.986691    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6xrxr\""
Sep 11 14:40:27.986858 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.986769    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 14:40:27.997694 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.996674    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w7klw"]
Sep 11 14:40:27.997694 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:27.997655    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 14:40:28.001125 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.001092    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b7f94f844-q7kdx"]
Sep 11 14:40:28.001255 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.001133    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pqq7j"]
Sep 11 14:40:28.001255 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.001215    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.003468 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.003427    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqq7j"
Sep 11 14:40:28.004565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.004542    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 14:40:28.004869 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.004851    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vx9kl\""
Sep 11 14:40:28.005028 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.005007    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 14:40:28.005970 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.005922    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 14:40:28.006061 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.006008    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 14:40:28.006312 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.006296    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tl5m6\""
Sep 11 14:40:28.006413 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.006398    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 14:40:28.015301 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.015270    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w7klw"]
Sep 11 14:40:28.016715 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.016653    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqq7j"]
Sep 11 14:40:28.050855 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.050822    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nm9\" (UniqueName: \"kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9\") pod \"network-check-target-9l6tg\" (UID: \"4bd24af6-f410-4833-816b-772fa4da84f1\") " pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:28.051017 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.050967    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:40:28.051017 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.050988    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:40:28.051017 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.050999    2559 projected.go:196] Error preparing data for projected volume kube-api-access-89nm9 for pod openshift-network-diagnostics/network-check-target-9l6tg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:28.051168 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.051060    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9 podName:4bd24af6-f410-4833-816b-772fa4da84f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:00.051041368 +0000 UTC m=+66.373669731 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-89nm9" (UniqueName: "kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9") pod "network-check-target-9l6tg" (UID: "4bd24af6-f410-4833-816b-772fa4da84f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:28.151339 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151300    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-config-volume\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.151525 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-ca-trust-extracted\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.151525 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-bound-sa-token\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.151525 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151416    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.151525 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151482    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-tmp-dir\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.151525 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmj9w\" (UniqueName: \"kubernetes.io/projected/1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238-kube-api-access-kmj9w\") pod \"ingress-canary-pqq7j\" (UID: \"1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238\") " pod="openshift-ingress-canary/ingress-canary-pqq7j"
Sep 11 14:40:28.151763 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-installation-pull-secrets\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.151763 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151615    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6klg\" (UniqueName: \"kubernetes.io/projected/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-kube-api-access-n6klg\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.151763 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238-cert\") pod \"ingress-canary-pqq7j\" (UID: \"1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238\") " pod="openshift-ingress-canary/ingress-canary-pqq7j"
Sep 11 14:40:28.151763 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151664    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.151763 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151710    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-certificates\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.151763 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151728    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbpq9\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-kube-api-access-lbpq9\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.151763 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-trusted-ca\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.152009 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.151784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-image-registry-private-configuration\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.162078 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.162047    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:28.251479 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.251436    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:28.252356 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238-cert\") pod \"ingress-canary-pqq7j\" (UID: \"1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238\") " pod="openshift-ingress-canary/ingress-canary-pqq7j"
Sep 11 14:40:28.252469 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252371    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.252469 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252422    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-certificates\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.252469 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbpq9\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-kube-api-access-lbpq9\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.252634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252493    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-trusted-ca\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.252634 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.252503    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:40:28.252634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-image-registry-private-configuration\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.252634 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.252576    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls podName:ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:28.752556989 +0000 UTC m=+35.075185355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls") pod "dns-default-w7klw" (UID: "ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c") : secret "dns-default-metrics-tls" not found
Sep 11 14:40:28.252634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-config-volume\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.252880 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-ca-trust-extracted\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.252880 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-bound-sa-token\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.252880 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.252880 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252711    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-tmp-dir\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.252880 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmj9w\" (UniqueName: \"kubernetes.io/projected/1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238-kube-api-access-kmj9w\") pod \"ingress-canary-pqq7j\" (UID: \"1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238\") " pod="openshift-ingress-canary/ingress-canary-pqq7j"
Sep 11 14:40:28.252880 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252770    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-installation-pull-secrets\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.252880 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.252811    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6klg\" (UniqueName: \"kubernetes.io/projected/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-kube-api-access-n6klg\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.252880 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.252866    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:40:28.252880 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.252882    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f94f844-q7kdx: secret "image-registry-tls" not found
Sep 11 14:40:28.253301 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.252938    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls podName:b5e9fa73-5376-4267-a1d3-90625b5f0fd6 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:28.752923463 +0000 UTC m=+35.075551811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls") pod "image-registry-5b7f94f844-q7kdx" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6") : secret "image-registry-tls" not found
Sep 11 14:40:28.253301 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.253042    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-certificates\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.253301 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.253177    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-config-volume\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.253301 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.253254    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-ca-trust-extracted\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.253481 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.253324    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-tmp-dir\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.253564 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.253545    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-trusted-ca\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.256597 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.256539    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238-cert\") pod \"ingress-canary-pqq7j\" (UID: \"1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238\") " pod="openshift-ingress-canary/ingress-canary-pqq7j"
Sep 11 14:40:28.256597 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.256574    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-image-registry-private-configuration\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.256597 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.256583    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-installation-pull-secrets\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.264528 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.264499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6klg\" (UniqueName: \"kubernetes.io/projected/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-kube-api-access-n6klg\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.264639 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.264544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-bound-sa-token\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.264872 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.264854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmj9w\" (UniqueName: \"kubernetes.io/projected/1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238-kube-api-access-kmj9w\") pod \"ingress-canary-pqq7j\" (UID: \"1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238\") " pod="openshift-ingress-canary/ingress-canary-pqq7j"
Sep 11 14:40:28.265033 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.265016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbpq9\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-kube-api-access-lbpq9\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.319297 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.319265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqq7j"
Sep 11 14:40:28.608973 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.608888    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqq7j"]
Sep 11 14:40:28.612753 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:40:28.612726    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ead5d7d_37b1_4bd4_a2c9_33d33d1a6238.slice/crio-43d590cef76a9e90386ee207ca6feab5b6df0ce15212f945843879cebfae3d2b WatchSource:0}: Error finding container 43d590cef76a9e90386ee207ca6feab5b6df0ce15212f945843879cebfae3d2b: Status 404 returned error can't find the container with id 43d590cef76a9e90386ee207ca6feab5b6df0ce15212f945843879cebfae3d2b
Sep 11 14:40:28.757475 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.757419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:28.757626 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:28.757508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:28.757626 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.757575    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:40:28.757626 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.757597    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f94f844-q7kdx: secret "image-registry-tls" not found
Sep 11 14:40:28.757626 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.757622    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:40:28.757817 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.757651    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls podName:b5e9fa73-5376-4267-a1d3-90625b5f0fd6 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:29.757635164 +0000 UTC m=+36.080263518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls") pod "image-registry-5b7f94f844-q7kdx" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6") : secret "image-registry-tls" not found
Sep 11 14:40:28.757817 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:28.757664    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls podName:ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:29.757658241 +0000 UTC m=+36.080286588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls") pod "dns-default-w7klw" (UID: "ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c") : secret "dns-default-metrics-tls" not found
Sep 11 14:40:29.173697 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.173660    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:29.174518 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.173660    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:40:29.174518 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.173660    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:40:29.178330 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.178294    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 14:40:29.178330 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.178312    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jjd7d\""
Sep 11 14:40:29.178562 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.178365    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 14:40:29.178562 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.178304    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 14:40:29.178562 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.178300    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 14:40:29.178562 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.178369    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xdw2d\""
Sep 11 14:40:29.444057 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.443974    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqq7j" event={"ID":"1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238","Type":"ContainerStarted","Data":"43d590cef76a9e90386ee207ca6feab5b6df0ce15212f945843879cebfae3d2b"}
Sep 11 14:40:29.765990 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.765945    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:29.766180 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:29.766033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:29.766180 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:29.766153    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:40:29.766180 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:29.766173    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f94f844-q7kdx: secret "image-registry-tls" not found
Sep 11 14:40:29.766297 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:29.766199    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:40:29.766297 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:29.766266    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls podName:b5e9fa73-5376-4267-a1d3-90625b5f0fd6 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:31.766241618 +0000 UTC m=+38.088869985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls") pod "image-registry-5b7f94f844-q7kdx" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6") : secret "image-registry-tls" not found
Sep 11 14:40:29.766377 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:29.766297    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls podName:ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:31.766280624 +0000 UTC m=+38.088908985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls") pod "dns-default-w7klw" (UID: "ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c") : secret "dns-default-metrics-tls" not found
Sep 11 14:40:30.703139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:30.703088    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:30.801735 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:30.801696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:31.640630 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:31.640594    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:31.742200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:31.742167    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:31.784968 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:31.784929    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:31.785154 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:31.785060    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:40:31.785154 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:31.785119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:31.785154 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:31.785135    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls podName:ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:35.785112998 +0000 UTC m=+42.107741349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls") pod "dns-default-w7klw" (UID: "ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c") : secret "dns-default-metrics-tls" not found
Sep 11 14:40:31.785284 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:31.785222    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:40:31.785284 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:31.785236    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f94f844-q7kdx: secret "image-registry-tls" not found
Sep 11 14:40:31.785348 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:31.785290    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls podName:b5e9fa73-5376-4267-a1d3-90625b5f0fd6 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:35.78527214 +0000 UTC m=+42.107900494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls") pod "image-registry-5b7f94f844-q7kdx" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6") : secret "image-registry-tls" not found
Sep 11 14:40:32.598751 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:32.598419    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:32.756890 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:32.756852    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:32.819687 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:32.819648    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:33.454992 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:33.454908    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqq7j" event={"ID":"1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238","Type":"ContainerStarted","Data":"e43560672e79e2f48839e8ed89c2599a4f42b762fc38ef61af092b9adde38e1f"}
Sep 11 14:40:33.457638 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:33.457608    2559 generic.go:356] "Generic (PLEG): container finished" podID="6bb6b12f-ef56-49af-a24e-9759831b31b7" containerID="95cc5fe91d5bc1a48543ee1516400e10827103df8e7c368e3312ebd4cc8d77b8" exitCode=0
Sep 11 14:40:33.457777 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:33.457664    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r5kp" event={"ID":"6bb6b12f-ef56-49af-a24e-9759831b31b7","Type":"ContainerDied","Data":"95cc5fe91d5bc1a48543ee1516400e10827103df8e7c368e3312ebd4cc8d77b8"}
Sep 11 14:40:33.473232 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:33.473178    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pqq7j" podStartSLOduration=2.672940609 podStartE2EDuration="6.473162797s" totalImagesPullingTime="3.800222188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:33.472264404 +0000 UTC m=+39.794892774" watchObservedRunningTime="2026-09-11 14:40:33.473162797 +0000 UTC m=+39.795791161"
Sep 11 14:40:34.463389 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:34.463352    2559 generic.go:356] "Generic (PLEG): container finished" podID="6bb6b12f-ef56-49af-a24e-9759831b31b7" containerID="52d2919ba9be4ce530b657b9435426b37f6414ddb8b053fa0e436987e78da0d4" exitCode=0
Sep 11 14:40:34.463865 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:34.463435    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r5kp" event={"ID":"6bb6b12f-ef56-49af-a24e-9759831b31b7","Type":"ContainerDied","Data":"52d2919ba9be4ce530b657b9435426b37f6414ddb8b053fa0e436987e78da0d4"}
Sep 11 14:40:35.469353 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:35.469318    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r5kp" event={"ID":"6bb6b12f-ef56-49af-a24e-9759831b31b7","Type":"ContainerStarted","Data":"299e71937e5a5ed79b047f8473c547241962bc3a459b49d39a515e79be0b7aa1"}
Sep 11 14:40:35.494410 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:35.494355    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8r5kp" podStartSLOduration=10.231299047 podStartE2EDuration="41.494337734s" totalImagesPullingTime="31.263038687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:35.492828214 +0000 UTC m=+41.815456583" watchObservedRunningTime="2026-09-11 14:40:35.494337734 +0000 UTC m=+41.816966104"
Sep 11 14:40:35.617012 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:35.616983    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:35.722162 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:35.722128    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:35.814926 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:35.814886    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:35.815085 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:35.814954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:35.815085 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:35.815040    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:40:35.815085 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:35.815053    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f94f844-q7kdx: secret "image-registry-tls" not found
Sep 11 14:40:35.815185 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:35.815108    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls podName:b5e9fa73-5376-4267-a1d3-90625b5f0fd6 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:43.815092802 +0000 UTC m=+50.137721155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls") pod "image-registry-5b7f94f844-q7kdx" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6") : secret "image-registry-tls" not found
Sep 11 14:40:35.815185 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:35.815106    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:40:35.815185 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:35.815161    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls podName:ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:43.815146945 +0000 UTC m=+50.137775292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls") pod "dns-default-w7klw" (UID: "ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c") : secret "dns-default-metrics-tls" not found
Sep 11 14:40:39.137366 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:39.137329    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:39.237089 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:39.237057    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:42.688371 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:42.688329    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:42.767328 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:42.767293    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:42.980391 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:42.980359    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:43.048434 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:43.048393    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:43.148471 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:43.148432    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:43.249747 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:43.249653    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:43.871311 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:43.871266    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:43.871768 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:43.871348    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:43.871768 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:43.871433    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:40:43.871768 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:43.871446    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f94f844-q7kdx: secret "image-registry-tls" not found
Sep 11 14:40:43.871768 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:43.871498    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:40:43.871768 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:43.871521    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls podName:b5e9fa73-5376-4267-a1d3-90625b5f0fd6 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:59.871504648 +0000 UTC m=+66.194133001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls") pod "image-registry-5b7f94f844-q7kdx" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6") : secret "image-registry-tls" not found
Sep 11 14:40:43.871768 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:43.871546    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls podName:ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:59.871532356 +0000 UTC m=+66.194160722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls") pod "dns-default-w7klw" (UID: "ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c") : secret "dns-default-metrics-tls" not found
Sep 11 14:40:44.020029 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:44.019993    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:44.121219 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:44.121183    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:47.687593 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:47.687558    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:47.788225 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:47.788193    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:47.851051 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:47.851022    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:47.924160 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:47.924130    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:54.677533 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:54.677495    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:54.737971 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:54.737935    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:55.439936 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:55.439906    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d6xtk"
Sep 11 14:40:55.714007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:55.713921    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:55.775508 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:55.775470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:56.057675 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.057645    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"]
Sep 11 14:40:56.059793 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.059774    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:56.063349 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.063324    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 11 14:40:56.064600 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.064576    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 11 14:40:56.064793 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.064650    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l752l\""
Sep 11 14:40:56.064793 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.064650    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 14:40:56.064793 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.064707    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 14:40:56.068744 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.068725    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-ctd5d"]
Sep 11 14:40:56.070534 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.070515    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.070682 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.070665    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"]
Sep 11 14:40:56.072052 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.072035    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"]
Sep 11 14:40:56.072179 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.072162    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"
Sep 11 14:40:56.073762 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.073742    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:40:56.075880 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.075863    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-67r22\""
Sep 11 14:40:56.075972 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.075948    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 14:40:56.076227 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.076205    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 14:40:56.076318 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.076239    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8z65r\""
Sep 11 14:40:56.076404 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.076386    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rhb6j\""
Sep 11 14:40:56.076601 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.076490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 11 14:40:56.076601 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.076516    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 11 14:40:56.076601 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.076528    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 11 14:40:56.076932 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.076789    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 14:40:56.076984 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.076932    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 14:40:56.076984 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.076959    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 14:40:56.077218 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.077204    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 14:40:56.077282 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.077232    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:40:56.077522 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.077506    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:40:56.083534 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.083511    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"]
Sep 11 14:40:56.087239 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.087217    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 14:40:56.094091 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.094059    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"]
Sep 11 14:40:56.094727 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.094706    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"]
Sep 11 14:40:56.095270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.095254    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-ctd5d"]
Sep 11 14:40:56.160282 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160247    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfvlp\" (UniqueName: \"kubernetes.io/projected/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-kube-api-access-cfvlp\") pod \"cluster-samples-operator-86499c89c4-qkf6g\" (UID: \"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:40:56.160282 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:56.160558 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160306    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b35287e9-9950-4b9d-af5c-969dd954fead-snapshots\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.160558 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b35287e9-9950-4b9d-af5c-969dd954fead-tmp\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.160558 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b35287e9-9950-4b9d-af5c-969dd954fead-service-ca-bundle\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.160558 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qkf6g\" (UID: \"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:40:56.160558 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160486    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c38c07c-2f0c-4e2a-a730-1280542e9fdc-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-hkdtk\" (UID: \"9c38c07c-2f0c-4e2a-a730-1280542e9fdc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"
Sep 11 14:40:56.160814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160558    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c38c07c-2f0c-4e2a-a730-1280542e9fdc-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-hkdtk\" (UID: \"9c38c07c-2f0c-4e2a-a730-1280542e9fdc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"
Sep 11 14:40:56.160814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5597d\" (UniqueName: \"kubernetes.io/projected/9c38c07c-2f0c-4e2a-a730-1280542e9fdc-kube-api-access-5597d\") pod \"kube-storage-version-migrator-operator-6c8f84f897-hkdtk\" (UID: \"9c38c07c-2f0c-4e2a-a730-1280542e9fdc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"
Sep 11 14:40:56.160814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160650    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/09522222-5124-4c7e-a196-39d46deeaa2e-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:56.160814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b35287e9-9950-4b9d-af5c-969dd954fead-trusted-ca-bundle\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.160814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160703    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfxtg\" (UniqueName: \"kubernetes.io/projected/b35287e9-9950-4b9d-af5c-969dd954fead-kube-api-access-kfxtg\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.160814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9rmk\" (UniqueName: \"kubernetes.io/projected/09522222-5124-4c7e-a196-39d46deeaa2e-kube-api-access-g9rmk\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:56.160814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.160772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b35287e9-9950-4b9d-af5c-969dd954fead-serving-cert\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.261926 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.261874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qkf6g\" (UID: \"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:40:56.261926 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.261930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c38c07c-2f0c-4e2a-a730-1280542e9fdc-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-hkdtk\" (UID: \"9c38c07c-2f0c-4e2a-a730-1280542e9fdc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"
Sep 11 14:40:56.262202 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.261974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c38c07c-2f0c-4e2a-a730-1280542e9fdc-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-hkdtk\" (UID: \"9c38c07c-2f0c-4e2a-a730-1280542e9fdc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"
Sep 11 14:40:56.262202 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262002    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5597d\" (UniqueName: \"kubernetes.io/projected/9c38c07c-2f0c-4e2a-a730-1280542e9fdc-kube-api-access-5597d\") pod \"kube-storage-version-migrator-operator-6c8f84f897-hkdtk\" (UID: \"9c38c07c-2f0c-4e2a-a730-1280542e9fdc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"
Sep 11 14:40:56.262202 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:56.262031    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 14:40:56.262202 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/09522222-5124-4c7e-a196-39d46deeaa2e-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:56.262202 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262064    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b35287e9-9950-4b9d-af5c-969dd954fead-trusted-ca-bundle\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.262202 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262086    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfxtg\" (UniqueName: \"kubernetes.io/projected/b35287e9-9950-4b9d-af5c-969dd954fead-kube-api-access-kfxtg\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.262202 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:56.262120    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls podName:afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:56.762098517 +0000 UTC m=+63.084726870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qkf6g" (UID: "afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2") : secret "samples-operator-tls" not found
Sep 11 14:40:56.262202 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262171    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9rmk\" (UniqueName: \"kubernetes.io/projected/09522222-5124-4c7e-a196-39d46deeaa2e-kube-api-access-g9rmk\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:56.262202 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b35287e9-9950-4b9d-af5c-969dd954fead-serving-cert\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.262668 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262257    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfvlp\" (UniqueName: \"kubernetes.io/projected/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-kube-api-access-cfvlp\") pod \"cluster-samples-operator-86499c89c4-qkf6g\" (UID: \"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:40:56.262668 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:56.262668 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262306    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b35287e9-9950-4b9d-af5c-969dd954fead-snapshots\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.262668 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b35287e9-9950-4b9d-af5c-969dd954fead-tmp\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.262668 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b35287e9-9950-4b9d-af5c-969dd954fead-service-ca-bundle\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.262668 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:56.262546    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 14:40:56.262668 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:56.262612    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls podName:09522222-5124-4c7e-a196-39d46deeaa2e nodeName:}" failed. No retries permitted until 2026-09-11 14:40:56.76259498 +0000 UTC m=+63.085223343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cp67f" (UID: "09522222-5124-4c7e-a196-39d46deeaa2e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 14:40:56.263021 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262724    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c38c07c-2f0c-4e2a-a730-1280542e9fdc-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-hkdtk\" (UID: \"9c38c07c-2f0c-4e2a-a730-1280542e9fdc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"
Sep 11 14:40:56.263021 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.262880    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/09522222-5124-4c7e-a196-39d46deeaa2e-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:56.263120 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.263052    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b35287e9-9950-4b9d-af5c-969dd954fead-service-ca-bundle\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.263120 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.263075    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b35287e9-9950-4b9d-af5c-969dd954fead-trusted-ca-bundle\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.263208 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.263116    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b35287e9-9950-4b9d-af5c-969dd954fead-snapshots\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.263253 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.263236    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b35287e9-9950-4b9d-af5c-969dd954fead-tmp\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.265252 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.265229    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c38c07c-2f0c-4e2a-a730-1280542e9fdc-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-hkdtk\" (UID: \"9c38c07c-2f0c-4e2a-a730-1280542e9fdc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"
Sep 11 14:40:56.265375 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.265358    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b35287e9-9950-4b9d-af5c-969dd954fead-serving-cert\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.277847 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.277824    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5597d\" (UniqueName: \"kubernetes.io/projected/9c38c07c-2f0c-4e2a-a730-1280542e9fdc-kube-api-access-5597d\") pod \"kube-storage-version-migrator-operator-6c8f84f897-hkdtk\" (UID: \"9c38c07c-2f0c-4e2a-a730-1280542e9fdc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"
Sep 11 14:40:56.279563 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.279538    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfxtg\" (UniqueName: \"kubernetes.io/projected/b35287e9-9950-4b9d-af5c-969dd954fead-kube-api-access-kfxtg\") pod \"insights-operator-5844b856db-ctd5d\" (UID: \"b35287e9-9950-4b9d-af5c-969dd954fead\") " pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.279789 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.279768    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfvlp\" (UniqueName: \"kubernetes.io/projected/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-kube-api-access-cfvlp\") pod \"cluster-samples-operator-86499c89c4-qkf6g\" (UID: \"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:40:56.279958 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.279938    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9rmk\" (UniqueName: \"kubernetes.io/projected/09522222-5124-4c7e-a196-39d46deeaa2e-kube-api-access-g9rmk\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:56.305242 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.305219    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:56.370885 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.370806    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:56.379553 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.379529    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-ctd5d"
Sep 11 14:40:56.388467 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.388226    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"
Sep 11 14:40:56.655421 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.655339    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-ctd5d"]
Sep 11 14:40:56.659528 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:40:56.659501    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35287e9_9950_4b9d_af5c_969dd954fead.slice/crio-88f3ba14f72a1d0a6bf4f4c7a3440d6774d65be59cedc467a929e28b76523b06 WatchSource:0}: Error finding container 88f3ba14f72a1d0a6bf4f4c7a3440d6774d65be59cedc467a929e28b76523b06: Status 404 returned error can't find the container with id 88f3ba14f72a1d0a6bf4f4c7a3440d6774d65be59cedc467a929e28b76523b06
Sep 11 14:40:56.675836 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.675805    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk"]
Sep 11 14:40:56.678627 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:40:56.678597    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c38c07c_2f0c_4e2a_a730_1280542e9fdc.slice/crio-14f8d87482e636d81ab0d26b0f767f5ae94dc81d7ae68fd64baaa325f0864ec1 WatchSource:0}: Error finding container 14f8d87482e636d81ab0d26b0f767f5ae94dc81d7ae68fd64baaa325f0864ec1: Status 404 returned error can't find the container with id 14f8d87482e636d81ab0d26b0f767f5ae94dc81d7ae68fd64baaa325f0864ec1
Sep 11 14:40:56.766800 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.766765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qkf6g\" (UID: \"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:40:56.767230 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:56.766890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:56.767230 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:56.766921    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 14:40:56.767230 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:56.766981    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls podName:afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:57.766966226 +0000 UTC m=+64.089594589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qkf6g" (UID: "afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2") : secret "samples-operator-tls" not found
Sep 11 14:40:56.767230 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:56.767031    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 14:40:56.767230 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:56.767091    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls podName:09522222-5124-4c7e-a196-39d46deeaa2e nodeName:}" failed. No retries permitted until 2026-09-11 14:40:57.767076441 +0000 UTC m=+64.089704790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cp67f" (UID: "09522222-5124-4c7e-a196-39d46deeaa2e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 14:40:57.524235 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:57.524192    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk" event={"ID":"9c38c07c-2f0c-4e2a-a730-1280542e9fdc","Type":"ContainerStarted","Data":"14f8d87482e636d81ab0d26b0f767f5ae94dc81d7ae68fd64baaa325f0864ec1"}
Sep 11 14:40:57.525704 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:57.525640    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-ctd5d" event={"ID":"b35287e9-9950-4b9d-af5c-969dd954fead","Type":"ContainerStarted","Data":"88f3ba14f72a1d0a6bf4f4c7a3440d6774d65be59cedc467a929e28b76523b06"}
Sep 11 14:40:57.776240 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:57.776150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qkf6g\" (UID: \"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:40:57.776704 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:57.776283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:57.776704 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:57.776415    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 14:40:57.776704 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:57.776488    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls podName:09522222-5124-4c7e-a196-39d46deeaa2e nodeName:}" failed. No retries permitted until 2026-09-11 14:40:59.776466282 +0000 UTC m=+66.099094645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cp67f" (UID: "09522222-5124-4c7e-a196-39d46deeaa2e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 14:40:57.776969 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:57.776876    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 14:40:57.776969 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:57.776932    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls podName:afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:59.776916215 +0000 UTC m=+66.099544565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qkf6g" (UID: "afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2") : secret "samples-operator-tls" not found
Sep 11 14:40:58.993583 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:58.993550    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-9pdfn"]
Sep 11 14:40:58.996401 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:58.996381    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-9pdfn"
Sep 11 14:40:58.998919 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:58.998897    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-h7c6h\""
Sep 11 14:40:59.005985 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.005964    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-9pdfn"]
Sep 11 14:40:59.089010 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.088987    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8zh\" (UniqueName: \"kubernetes.io/projected/089e951d-07b4-48ec-a96b-76ce83bfdb17-kube-api-access-cb8zh\") pod \"network-check-source-6f8b768455-9pdfn\" (UID: \"089e951d-07b4-48ec-a96b-76ce83bfdb17\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-9pdfn"
Sep 11 14:40:59.189799 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.189762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8zh\" (UniqueName: \"kubernetes.io/projected/089e951d-07b4-48ec-a96b-76ce83bfdb17-kube-api-access-cb8zh\") pod \"network-check-source-6f8b768455-9pdfn\" (UID: \"089e951d-07b4-48ec-a96b-76ce83bfdb17\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-9pdfn"
Sep 11 14:40:59.199343 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.199319    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 14:40:59.209595 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.209576    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 14:40:59.220141 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.220116    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8zh\" (UniqueName: \"kubernetes.io/projected/089e951d-07b4-48ec-a96b-76ce83bfdb17-kube-api-access-cb8zh\") pod \"network-check-source-6f8b768455-9pdfn\" (UID: \"089e951d-07b4-48ec-a96b-76ce83bfdb17\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-9pdfn"
Sep 11 14:40:59.305391 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.305289    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-9pdfn"
Sep 11 14:40:59.532829 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.532784    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk" event={"ID":"9c38c07c-2f0c-4e2a-a730-1280542e9fdc","Type":"ContainerStarted","Data":"46f1a70084aafc9e48f3f08516f5a5b0c38cfc77782efc0d022441d11bbf94db"}
Sep 11 14:40:59.534309 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.534281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-ctd5d" event={"ID":"b35287e9-9950-4b9d-af5c-969dd954fead","Type":"ContainerStarted","Data":"8e776083509e1e05204c4f232a452645cff2f5ccca221b6cc5b3b5a3789c9160"}
Sep 11 14:40:59.552515 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.552447    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk" podStartSLOduration=1.180908982 podStartE2EDuration="3.552431523s" totalImagesPullingTime="2.371522541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:59.551016011 +0000 UTC m=+65.873644392" watchObservedRunningTime="2026-09-11 14:40:59.552431523 +0000 UTC m=+65.875059914"
Sep 11 14:40:59.568218 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.568115    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-ctd5d" podStartSLOduration=1.181330889 podStartE2EDuration="3.568094762s" totalImagesPullingTime="2.386763873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:59.567413682 +0000 UTC m=+65.890042053" watchObservedRunningTime="2026-09-11 14:40:59.568094762 +0000 UTC m=+65.890723135"
Sep 11 14:40:59.581665 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.581634    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-9pdfn"]
Sep 11 14:40:59.585173 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:40:59.585146    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089e951d_07b4_48ec_a96b_76ce83bfdb17.slice/crio-f1ac61b06f5c99928b63c458c7553ed53f15601fb16fd836b340b5045357d0e6 WatchSource:0}: Error finding container f1ac61b06f5c99928b63c458c7553ed53f15601fb16fd836b340b5045357d0e6: Status 404 returned error can't find the container with id f1ac61b06f5c99928b63c458c7553ed53f15601fb16fd836b340b5045357d0e6
Sep 11 14:40:59.797613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.797552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:40:59.797613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.797622    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qkf6g\" (UID: \"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:40:59.797880 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:59.797754    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 14:40:59.797880 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:59.797776    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 14:40:59.797880 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:59.797841    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls podName:09522222-5124-4c7e-a196-39d46deeaa2e nodeName:}" failed. No retries permitted until 2026-09-11 14:41:03.79781954 +0000 UTC m=+70.120447903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cp67f" (UID: "09522222-5124-4c7e-a196-39d46deeaa2e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 14:40:59.797880 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:59.797862    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls podName:afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:03.797851848 +0000 UTC m=+70.120480225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qkf6g" (UID: "afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2") : secret "samples-operator-tls" not found
Sep 11 14:40:59.898682 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.898591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:40:59.898682 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.898627    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:59.898909 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.898705    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:40:59.898909 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:59.898761    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:40:59.898909 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:59.898787    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f94f844-q7kdx: secret "image-registry-tls" not found
Sep 11 14:40:59.898909 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:59.898788    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:40:59.898909 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:59.898844    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls podName:ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c nodeName:}" failed. No retries permitted until 2026-09-11 14:41:31.898829999 +0000 UTC m=+98.221458351 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls") pod "dns-default-w7klw" (UID: "ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c") : secret "dns-default-metrics-tls" not found
Sep 11 14:40:59.898909 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:40:59.898857    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls podName:b5e9fa73-5376-4267-a1d3-90625b5f0fd6 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:31.89885077 +0000 UTC m=+98.221479118 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls") pod "image-registry-5b7f94f844-q7kdx" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6") : secret "image-registry-tls" not found
Sep 11 14:40:59.901613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.901595    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 14:40:59.911857 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.911836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c89e9a20-c4e4-4c03-9601-8e47f2ad2df2-original-pull-secret\") pod \"global-pull-secret-syncer-24976\" (UID: \"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2\") " pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:40:59.922584 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.922555    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:40:59.999154 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:40:59.999122    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:41:00.002353 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.002330    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 14:41:00.009562 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:00.009544    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 14:41:00.009653 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:00.009603    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs podName:fde006ea-2cba-4b23-b83e-6f7703193a31 nodeName:}" failed. No retries permitted until 2026-09-11 14:42:04.009585679 +0000 UTC m=+130.332214030 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs") pod "network-metrics-daemon-2snh4" (UID: "fde006ea-2cba-4b23-b83e-6f7703193a31") : secret "metrics-daemon-secret" not found
Sep 11 14:41:00.023316 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.023272    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:00.086071 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.086035    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24976"
Sep 11 14:41:00.100366 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.100333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nm9\" (UniqueName: \"kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9\") pod \"network-check-target-9l6tg\" (UID: \"4bd24af6-f410-4833-816b-772fa4da84f1\") " pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:41:00.103528 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.103501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nm9\" (UniqueName: \"kubernetes.io/projected/4bd24af6-f410-4833-816b-772fa4da84f1-kube-api-access-89nm9\") pod \"network-check-target-9l6tg\" (UID: \"4bd24af6-f410-4833-816b-772fa4da84f1\") " pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:41:00.372507 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.372472    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-24976"]
Sep 11 14:41:00.376361 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:00.376324    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89e9a20_c4e4_4c03_9601_8e47f2ad2df2.slice/crio-d1d9f958556ff7c8e1e46ac52bb441b03968eb790ff5ac244020472085812ba1 WatchSource:0}: Error finding container d1d9f958556ff7c8e1e46ac52bb441b03968eb790ff5ac244020472085812ba1: Status 404 returned error can't find the container with id d1d9f958556ff7c8e1e46ac52bb441b03968eb790ff5ac244020472085812ba1
Sep 11 14:41:00.401715 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.401621    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xdw2d\""
Sep 11 14:41:00.409757 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.409706    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:41:00.538672 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.538584    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-9pdfn" event={"ID":"089e951d-07b4-48ec-a96b-76ce83bfdb17","Type":"ContainerStarted","Data":"f1ac61b06f5c99928b63c458c7553ed53f15601fb16fd836b340b5045357d0e6"}
Sep 11 14:41:00.540301 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.539969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-24976" event={"ID":"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2","Type":"ContainerStarted","Data":"d1d9f958556ff7c8e1e46ac52bb441b03968eb790ff5ac244020472085812ba1"}
Sep 11 14:41:00.693247 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.693149    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9l6tg"]
Sep 11 14:41:00.697953 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:00.697898    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd24af6_f410_4833_816b_772fa4da84f1.slice/crio-c149aaa99fbae77ba8f6d0ae791f241198360f2c2ebc862fb0139a50b735cd3f WatchSource:0}: Error finding container c149aaa99fbae77ba8f6d0ae791f241198360f2c2ebc862fb0139a50b735cd3f: Status 404 returned error can't find the container with id c149aaa99fbae77ba8f6d0ae791f241198360f2c2ebc862fb0139a50b735cd3f
Sep 11 14:41:00.948147 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.948065    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:00.957387 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.957360    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6b6dq"]
Sep 11 14:41:00.962295 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.962279    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:00.965023 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.964975    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 14:41:00.965023 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.964975    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 14:41:00.965214 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.965166    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6tvzn\""
Sep 11 14:41:00.973965 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:00.973942    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6b6dq"]
Sep 11 14:41:01.005551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.005511    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-data-volume\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.006008 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.005572    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-494t2\" (UniqueName: \"kubernetes.io/projected/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-api-access-494t2\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.006008 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.005600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.006008 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.005653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-crio-socket\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.006008 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.005681    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.069490 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.069388    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:01.106842 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.106799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.107020 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.106914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-data-volume\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.107020 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.106948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-494t2\" (UniqueName: \"kubernetes.io/projected/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-api-access-494t2\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.107020 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:01.106964    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:01.107020 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.106989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.107204 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:01.107044    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls podName:2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf nodeName:}" failed. No retries permitted until 2026-09-11 14:41:01.607021809 +0000 UTC m=+67.929650171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6b6dq" (UID: "2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:01.107204 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:01.107087    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:41:01.107204 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.107099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-crio-socket\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.107204 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:01.107144    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm podName:2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf nodeName:}" failed. No retries permitted until 2026-09-11 14:41:01.607128869 +0000 UTC m=+67.929757219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6b6dq" (UID: "2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf") : configmap "kube-rbac-proxy" not found
Sep 11 14:41:01.107204 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.107196    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-crio-socket\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.107710 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.107299    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-data-volume\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.119084 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.119048    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-494t2\" (UniqueName: \"kubernetes.io/projected/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-api-access-494t2\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.138271 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.138235    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:01.545735 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.545699    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9l6tg" event={"ID":"4bd24af6-f410-4833-816b-772fa4da84f1","Type":"ContainerStarted","Data":"c149aaa99fbae77ba8f6d0ae791f241198360f2c2ebc862fb0139a50b735cd3f"}
Sep 11 14:41:01.552405 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.552373    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:01.611132 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.611062    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.611330 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.611146    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:01.611407 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:01.611331    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:01.611407 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:01.611395    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls podName:2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf nodeName:}" failed. No retries permitted until 2026-09-11 14:41:02.611375353 +0000 UTC m=+68.934003702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6b6dq" (UID: "2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:01.611651 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:01.611624    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:41:01.611817 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:01.611691    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm podName:2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf nodeName:}" failed. No retries permitted until 2026-09-11 14:41:02.611674751 +0000 UTC m=+68.934303116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6b6dq" (UID: "2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf") : configmap "kube-rbac-proxy" not found
Sep 11 14:41:01.639762 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.639726    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:01.817698 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.817614    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:01.912876 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:01.912774    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:02.101179 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:02.101085    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:02.190539 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:02.190476    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:02.267111 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:02.267063    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:02.603327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:02.603292    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7lxrw_7b7d0ed0-8337-4b8c-adfa-4918f5150593/dns-node-resolver/0.log"
Sep 11 14:41:02.618782 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:02.618746    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:02.618969 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:02.618856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:02.618969 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:02.618882    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:41:02.618969 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:02.618964    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm podName:2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf nodeName:}" failed. No retries permitted until 2026-09-11 14:41:04.618939113 +0000 UTC m=+70.941567489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6b6dq" (UID: "2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf") : configmap "kube-rbac-proxy" not found
Sep 11 14:41:02.618969 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:02.618969    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:02.619158 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:02.619005    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls podName:2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf nodeName:}" failed. No retries permitted until 2026-09-11 14:41:04.61899435 +0000 UTC m=+70.941622702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6b6dq" (UID: "2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:03.452986 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:03.452947    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:03.511886 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:03.511842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:03.803687 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:03.803656    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6r78k_c9533b2d-a936-407d-ad6f-e3480bcdd109/node-ca/0.log"
Sep 11 14:41:03.832168 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:03.832124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:41:03.832334 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:03.832184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qkf6g\" (UID: \"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:41:03.832334 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:03.832323    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 14:41:03.832467 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:03.832356    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 14:41:03.832467 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:03.832406    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls podName:09522222-5124-4c7e-a196-39d46deeaa2e nodeName:}" failed. No retries permitted until 2026-09-11 14:41:11.832384459 +0000 UTC m=+78.155012814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cp67f" (UID: "09522222-5124-4c7e-a196-39d46deeaa2e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 14:41:03.832467 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:03.832430    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls podName:afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:11.832419165 +0000 UTC m=+78.155047518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qkf6g" (UID: "afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2") : secret "samples-operator-tls" not found
Sep 11 14:41:04.442508 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:04.442450    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:04.639235 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:04.639193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:04.639687 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:04.639291    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:41:04.639687 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:04.639293    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:04.639687 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:04.639388    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm podName:2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf nodeName:}" failed. No retries permitted until 2026-09-11 14:41:08.639365308 +0000 UTC m=+74.961993659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6b6dq" (UID: "2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf") : configmap "kube-rbac-proxy" not found
Sep 11 14:41:04.639687 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:04.639397    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:04.639687 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:04.639474    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls podName:2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf nodeName:}" failed. No retries permitted until 2026-09-11 14:41:08.639443261 +0000 UTC m=+74.962071612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6b6dq" (UID: "2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:04.810424 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:04.810398    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pqq7j_1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238/serve-healthcheck-canary/0.log"
Sep 11 14:41:05.561386 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:05.561347    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-9pdfn" event={"ID":"089e951d-07b4-48ec-a96b-76ce83bfdb17","Type":"ContainerStarted","Data":"fa5839ebdcfaf30dd84916a5ea452e054d7c2013ccdad50ba0cefdab27ad0007"}
Sep 11 14:41:05.562944 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:05.562914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-24976" event={"ID":"c89e9a20-c4e4-4c03-9601-8e47f2ad2df2","Type":"ContainerStarted","Data":"14f27393dcb996043fb39a71a41436b5058ef83af5a72caf52a6d0169372d060"}
Sep 11 14:41:05.564391 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:05.564365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9l6tg" event={"ID":"4bd24af6-f410-4833-816b-772fa4da84f1","Type":"ContainerStarted","Data":"b628dd822075f71d7794663462fa92d6751cd4f7d23e823986b0664a6ee66065"}
Sep 11 14:41:05.564509 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:05.564496    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:41:05.582807 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:05.582762    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-9pdfn" podStartSLOduration=2.370885814 podStartE2EDuration="7.582748987s" totalImagesPullingTime="5.211863173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:05.582041974 +0000 UTC m=+71.904670346" watchObservedRunningTime="2026-09-11 14:41:05.582748987 +0000 UTC m=+71.905377402"
Sep 11 14:41:05.598684 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:05.598647    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-24976" podStartSLOduration=67.119651878 podStartE2EDuration="1m11.598634827s" totalImagesPullingTime="4.478982949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:05.598014363 +0000 UTC m=+71.920642734" watchObservedRunningTime="2026-09-11 14:41:05.598634827 +0000 UTC m=+71.921263197"
Sep 11 14:41:05.615570 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:05.615518    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9l6tg" podStartSLOduration=67.506963215 podStartE2EDuration="1m11.615500033s" totalImagesPullingTime="4.108536818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:05.615361781 +0000 UTC m=+71.937990151" watchObservedRunningTime="2026-09-11 14:41:05.615500033 +0000 UTC m=+71.938128404"
Sep 11 14:41:06.214022 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:06.213984    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:06.334051 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:06.334008    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:06.457200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:06.457167    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:06.605497 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:06.605443    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:06.705565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:06.705532    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:06.808404 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:06.808364    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:07.110390 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:07.110349    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:07.520920 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:07.520868    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:07.912230 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:07.912137    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:08.522802 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:08.522763    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:08.670985 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:08.670948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:08.671146 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:08.671005    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:08.671146 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:08.671062    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:41:08.671146 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:08.671124    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm podName:2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf nodeName:}" failed. No retries permitted until 2026-09-11 14:41:16.671106413 +0000 UTC m=+82.993734765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6b6dq" (UID: "2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf") : configmap "kube-rbac-proxy" not found
Sep 11 14:41:08.671146 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:08.671127    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:08.671278 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:08.671174    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls podName:2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf nodeName:}" failed. No retries permitted until 2026-09-11 14:41:16.671163063 +0000 UTC m=+82.993791411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6b6dq" (UID: "2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:09.105968 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:09.105928    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:09.308797 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:09.308757    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:09.508026 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:09.507987    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:09.907210 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:09.907115    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:10.271588 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:10.271541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:10.363724 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:10.363693    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:10.514448 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:10.514417    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:11.107593 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:11.107556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:11.168266 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:11.168236    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:11.303088 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:11.303030    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:11.713506 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:11.713472    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:11.897554 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:11.897524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:41:11.897714 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:11.897568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qkf6g\" (UID: \"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:41:11.897714 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:11.897666    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 14:41:11.897789 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:11.897726    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls podName:09522222-5124-4c7e-a196-39d46deeaa2e nodeName:}" failed. No retries permitted until 2026-09-11 14:41:27.89771073 +0000 UTC m=+94.220339077 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cp67f" (UID: "09522222-5124-4c7e-a196-39d46deeaa2e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 14:41:11.899882 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:11.899853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qkf6g\" (UID: \"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:41:11.908357 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:11.908318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:11.992251 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:11.992222    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"
Sep 11 14:41:12.273444 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:12.272385    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g"]
Sep 11 14:41:12.312540 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:12.312514    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:12.593499 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:12.593410    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g" event={"ID":"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2","Type":"ContainerStarted","Data":"34c345603dd2453bf301db6e17af7ad106d4f01a8507e1d5079cbc729e5859f0"}
Sep 11 14:41:12.914843 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:12.914759    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:13.505303 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:13.505267    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:13.567122 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:13.567088    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:13.704140 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:13.704104    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:13.901790 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:13.901692    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:14.080712 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:14.080677    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:14.307648 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:14.307613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:14.601014 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:14.600974    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g" event={"ID":"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2","Type":"ContainerStarted","Data":"e8c1d39196d51bc57436395c1a2b9fc187dc0273a8a3a1ffe75eca0f226a10b1"}
Sep 11 14:41:14.601410 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:14.601034    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g" event={"ID":"afb8fd7a-90bb-4173-89de-fc4b1dc2a7d2","Type":"ContainerStarted","Data":"feb9274faa0bae7de2006bca9ddb0db0c7bf84c8f3e69b47799171f909d329da"}
Sep 11 14:41:14.619344 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:14.619268    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qkf6g" podStartSLOduration=16.796976704 podStartE2EDuration="18.619251511s" totalImagesPullingTime="1.822274807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:14.618730128 +0000 UTC m=+80.941358498" watchObservedRunningTime="2026-09-11 14:41:14.619251511 +0000 UTC m=+80.941879882"
Sep 11 14:41:14.908796 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:14.908698    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:15.320859 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:15.320806    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:15.710950 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:15.710841    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:15.907129 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:15.907100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:16.302961 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:16.302929    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:16.714891 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:16.714703    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:16.738843 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:16.738810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:16.739010 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:16.738869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:16.739367 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:16.739335    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:16.741175 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:16.741151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6b6dq\" (UID: \"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf\") " pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:16.871227 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:16.871192    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6b6dq"
Sep 11 14:41:16.922853 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:16.922806    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:17.143194 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:17.143162    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6b6dq"]
Sep 11 14:41:17.146268 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:17.146239    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5e618a_1e46_4078_a3b2_8c2f97c4b8bf.slice/crio-91b7d2b2751dba9ea906a183db7b5348c1447793b006f027cc948ad9e7ae12ae WatchSource:0}: Error finding container 91b7d2b2751dba9ea906a183db7b5348c1447793b006f027cc948ad9e7ae12ae: Status 404 returned error can't find the container with id 91b7d2b2751dba9ea906a183db7b5348c1447793b006f027cc948ad9e7ae12ae
Sep 11 14:41:17.610800 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:17.610766    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6b6dq" event={"ID":"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf","Type":"ContainerStarted","Data":"ac2eede044683e29e18b8b98278a93c8ba402c529896dd9ab6c0a8813b3c2911"}
Sep 11 14:41:17.610968 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:17.610826    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6b6dq" event={"ID":"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf","Type":"ContainerStarted","Data":"91b7d2b2751dba9ea906a183db7b5348c1447793b006f027cc948ad9e7ae12ae"}
Sep 11 14:41:18.617056 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:18.617009    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6b6dq" event={"ID":"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf","Type":"ContainerStarted","Data":"b1cc5c373c9aa36597d564d0dada3359f8b59bf1478ecf0db827af49a2e2a4a4"}
Sep 11 14:41:19.622345 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:19.622306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6b6dq" event={"ID":"2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf","Type":"ContainerStarted","Data":"a722a7e5c5a9c94140f1818cd001b8e55730dbffac0ab316f088cff196870de2"}
Sep 11 14:41:19.644019 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:19.643964    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6b6dq" podStartSLOduration=17.457949377 podStartE2EDuration="19.643947189s" totalImagesPullingTime="2.185997812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:41:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:19.642954726 +0000 UTC m=+85.965583108" watchObservedRunningTime="2026-09-11 14:41:19.643947189 +0000 UTC m=+85.966575558"
Sep 11 14:41:21.521399 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:21.521367    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:21.900525 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:21.900413    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:21.994852 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:21.994813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:22.060101 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:22.060063    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:22.392289 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:22.392251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:22.559234 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:22.559191    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:22.624677 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:22.624639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:22.762761 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:22.762730    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:22.826291 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:22.826257    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:23.373937 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:23.373904    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:23.776180 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:23.776148    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:23.878672 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:23.878640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:23.992206 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:23.992174    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:24.126400 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:24.126315    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:24.227364 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:24.227328    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:27.660197 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.660164    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"]
Sep 11 14:41:27.662905 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.662888    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:27.665946 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.665917    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 14:41:27.666080 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.665978    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 14:41:27.667648 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.667625    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 14:41:27.667975 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.667649    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 14:41:27.689331 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.689295    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"]
Sep 11 14:41:27.766129 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.766065    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:27.828582 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.828544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e2a8eca-034a-4e99-8511-2f19ebfaf930-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d9649dc55-sl4ns\" (UID: \"0e2a8eca-034a-4e99-8511-2f19ebfaf930\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:27.828582 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.828581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e2a8eca-034a-4e99-8511-2f19ebfaf930-tmp\") pod \"klusterlet-addon-workmgr-5d9649dc55-sl4ns\" (UID: \"0e2a8eca-034a-4e99-8511-2f19ebfaf930\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:27.828788 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.828607    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks6dt\" (UniqueName: \"kubernetes.io/projected/0e2a8eca-034a-4e99-8511-2f19ebfaf930-kube-api-access-ks6dt\") pod \"klusterlet-addon-workmgr-5d9649dc55-sl4ns\" (UID: \"0e2a8eca-034a-4e99-8511-2f19ebfaf930\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:27.897190 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.897155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:27.929906 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.929829    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:41:27.929906 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.929867    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e2a8eca-034a-4e99-8511-2f19ebfaf930-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d9649dc55-sl4ns\" (UID: \"0e2a8eca-034a-4e99-8511-2f19ebfaf930\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:27.929906 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.929887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e2a8eca-034a-4e99-8511-2f19ebfaf930-tmp\") pod \"klusterlet-addon-workmgr-5d9649dc55-sl4ns\" (UID: \"0e2a8eca-034a-4e99-8511-2f19ebfaf930\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:27.929906 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.929908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6dt\" (UniqueName: \"kubernetes.io/projected/0e2a8eca-034a-4e99-8511-2f19ebfaf930-kube-api-access-ks6dt\") pod \"klusterlet-addon-workmgr-5d9649dc55-sl4ns\" (UID: \"0e2a8eca-034a-4e99-8511-2f19ebfaf930\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:27.930412 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.930381    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e2a8eca-034a-4e99-8511-2f19ebfaf930-tmp\") pod \"klusterlet-addon-workmgr-5d9649dc55-sl4ns\" (UID: \"0e2a8eca-034a-4e99-8511-2f19ebfaf930\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:27.932406 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.932382    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/09522222-5124-4c7e-a196-39d46deeaa2e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cp67f\" (UID: \"09522222-5124-4c7e-a196-39d46deeaa2e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:41:27.932503 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.932410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e2a8eca-034a-4e99-8511-2f19ebfaf930-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d9649dc55-sl4ns\" (UID: \"0e2a8eca-034a-4e99-8511-2f19ebfaf930\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:27.941340 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.941312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6dt\" (UniqueName: \"kubernetes.io/projected/0e2a8eca-034a-4e99-8511-2f19ebfaf930-kube-api-access-ks6dt\") pod \"klusterlet-addon-workmgr-5d9649dc55-sl4ns\" (UID: \"0e2a8eca-034a-4e99-8511-2f19ebfaf930\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:27.974812 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:27.974780    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:28.002787 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:28.002749    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:28.099069 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:28.099038    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:28.169217 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:28.169157    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"
Sep 11 14:41:28.252988 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:28.252960    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"]
Sep 11 14:41:28.255767 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:28.255741    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2a8eca_034a_4e99_8511_2f19ebfaf930.slice/crio-a5e792ef8b112ecd425575ea646d173cd1d5befb8ace45e55d18ed8bd5e5459f WatchSource:0}: Error finding container a5e792ef8b112ecd425575ea646d173cd1d5befb8ace45e55d18ed8bd5e5459f: Status 404 returned error can't find the container with id a5e792ef8b112ecd425575ea646d173cd1d5befb8ace45e55d18ed8bd5e5459f
Sep 11 14:41:28.447196 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:28.447164    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f"]
Sep 11 14:41:28.449885 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:28.449859    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09522222_5124_4c7e_a196_39d46deeaa2e.slice/crio-105a0cab78336373b7d776db0c44e96ea81421b1917a9253d7f72c13468e0d62 WatchSource:0}: Error finding container 105a0cab78336373b7d776db0c44e96ea81421b1917a9253d7f72c13468e0d62: Status 404 returned error can't find the container with id 105a0cab78336373b7d776db0c44e96ea81421b1917a9253d7f72c13468e0d62
Sep 11 14:41:28.649974 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:28.649938    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns" event={"ID":"0e2a8eca-034a-4e99-8511-2f19ebfaf930","Type":"ContainerStarted","Data":"a5e792ef8b112ecd425575ea646d173cd1d5befb8ace45e55d18ed8bd5e5459f"}
Sep 11 14:41:28.653209 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:28.653177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f" event={"ID":"09522222-5124-4c7e-a196-39d46deeaa2e","Type":"ContainerStarted","Data":"105a0cab78336373b7d776db0c44e96ea81421b1917a9253d7f72c13468e0d62"}
Sep 11 14:41:29.190612 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:29.190575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:29.267651 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:29.267582    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:29.334629 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:29.334555    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:31.552734 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.552694    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-798c4c6c45-zl7zn"]
Sep 11 14:41:31.555684 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.555659    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.558508 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.558487    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 14:41:31.558611 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.558571    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 14:41:31.558611 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.558602    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8v4w7\""
Sep 11 14:41:31.559945 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.559927    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 14:41:31.560047 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.559934    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 14:41:31.560047 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.560019    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 14:41:31.560162 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.560139    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 14:41:31.560222 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.560195    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 14:41:31.565554 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.565534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-798c4c6c45-zl7zn"]
Sep 11 14:41:31.665319 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.665284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-config\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.665510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.665328    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzsp5\" (UniqueName: \"kubernetes.io/projected/4d2deea2-5322-4201-b2d4-08c50d4fb76e-kube-api-access-fzsp5\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.665510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.665425    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-service-ca\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.665650 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.665514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-oauth-config\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.665650 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.665561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-oauth-serving-cert\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.665650 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.665619    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-serving-cert\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.766857 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.766820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-oauth-config\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.767060 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.766872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-oauth-serving-cert\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.767060 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.766941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-serving-cert\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.767060 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.766993    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-config\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.767060 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.767022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzsp5\" (UniqueName: \"kubernetes.io/projected/4d2deea2-5322-4201-b2d4-08c50d4fb76e-kube-api-access-fzsp5\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.767267 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.767164    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-service-ca\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.767747 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.767698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-oauth-serving-cert\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.767747 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.767726    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-config\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.767931 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.767852    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-service-ca\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.769709 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.769687    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-serving-cert\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.769817 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.769772    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-oauth-config\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.776076 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.776053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzsp5\" (UniqueName: \"kubernetes.io/projected/4d2deea2-5322-4201-b2d4-08c50d4fb76e-kube-api-access-fzsp5\") pod \"console-798c4c6c45-zl7zn\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") " pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.867129 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.867050    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:31.969523 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.969486    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:41:31.969701 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.969547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:41:31.972205 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.972180    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c-metrics-tls\") pod \"dns-default-w7klw\" (UID: \"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c\") " pod="openshift-dns/dns-default-w7klw"
Sep 11 14:41:31.972336 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:31.972265    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") pod \"image-registry-5b7f94f844-q7kdx\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") " pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:41:32.199305 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.199223    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6xrxr\""
Sep 11 14:41:32.206852 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.206819    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:41:32.217551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.217058    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vx9kl\""
Sep 11 14:41:32.223420 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.223398    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7klw"
Sep 11 14:41:32.562004 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.561945    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-798c4c6c45-zl7zn"]
Sep 11 14:41:32.565139 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:32.565073    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2deea2_5322_4201_b2d4_08c50d4fb76e.slice/crio-a9dec956d089bc0bb126473fb9005c5784264b76b8cff078356abb04d0333d3d WatchSource:0}: Error finding container a9dec956d089bc0bb126473fb9005c5784264b76b8cff078356abb04d0333d3d: Status 404 returned error can't find the container with id a9dec956d089bc0bb126473fb9005c5784264b76b8cff078356abb04d0333d3d
Sep 11 14:41:32.565741 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.565716    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b7f94f844-q7kdx"]
Sep 11 14:41:32.568759 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:32.568737    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e9fa73_5376_4267_a1d3_90625b5f0fd6.slice/crio-4a891215cfbf180d9ea539b4137f17d0c9c8904c83af0db8634b492d755f87b6 WatchSource:0}: Error finding container 4a891215cfbf180d9ea539b4137f17d0c9c8904c83af0db8634b492d755f87b6: Status 404 returned error can't find the container with id 4a891215cfbf180d9ea539b4137f17d0c9c8904c83af0db8634b492d755f87b6
Sep 11 14:41:32.582804 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.582767    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w7klw"]
Sep 11 14:41:32.585116 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:32.585090    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbd73b9_8f42_4afd_aa4d_38ae7f29ae9c.slice/crio-8fe2365c329619a7f76d5abf2eeea4d6e33a099254a5b56885fc5cca03e5b15d WatchSource:0}: Error finding container 8fe2365c329619a7f76d5abf2eeea4d6e33a099254a5b56885fc5cca03e5b15d: Status 404 returned error can't find the container with id 8fe2365c329619a7f76d5abf2eeea4d6e33a099254a5b56885fc5cca03e5b15d
Sep 11 14:41:32.653926 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.653888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:32.670944 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.670906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7klw" event={"ID":"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c","Type":"ContainerStarted","Data":"8fe2365c329619a7f76d5abf2eeea4d6e33a099254a5b56885fc5cca03e5b15d"}
Sep 11 14:41:32.672732 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.672704    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f" event={"ID":"09522222-5124-4c7e-a196-39d46deeaa2e","Type":"ContainerStarted","Data":"077d726cbc89807b55f1bbc2a5fd0630672cfc994d7f43663e34da9982ae03bb"}
Sep 11 14:41:32.674606 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.674582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns" event={"ID":"0e2a8eca-034a-4e99-8511-2f19ebfaf930","Type":"ContainerStarted","Data":"21011b99cbcf57cb8e8bc59e308dcb3ce65430c6ef7fddd9e0582c0b4921a0ce"}
Sep 11 14:41:32.674808 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.674769    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:32.676739 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.676614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx" event={"ID":"b5e9fa73-5376-4267-a1d3-90625b5f0fd6","Type":"ContainerStarted","Data":"81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71"}
Sep 11 14:41:32.676739 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.676664    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx" event={"ID":"b5e9fa73-5376-4267-a1d3-90625b5f0fd6","Type":"ContainerStarted","Data":"4a891215cfbf180d9ea539b4137f17d0c9c8904c83af0db8634b492d755f87b6"}
Sep 11 14:41:32.676739 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.676692    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:41:32.676952 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.676876    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns"
Sep 11 14:41:32.678055 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.678034    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798c4c6c45-zl7zn" event={"ID":"4d2deea2-5322-4201-b2d4-08c50d4fb76e","Type":"ContainerStarted","Data":"a9dec956d089bc0bb126473fb9005c5784264b76b8cff078356abb04d0333d3d"}
Sep 11 14:41:32.702962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.702906    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cp67f" podStartSLOduration=32.862089987 podStartE2EDuration="36.702887694s" totalImagesPullingTime="3.840797707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:32.702402852 +0000 UTC m=+99.025031224" watchObservedRunningTime="2026-09-11 14:41:32.702887694 +0000 UTC m=+99.025516065"
Sep 11 14:41:32.736690 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.736640    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx" podStartSLOduration=98.736624939 podStartE2EDuration="1m38.736624939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:32.736606524 +0000 UTC m=+99.059234894" watchObservedRunningTime="2026-09-11 14:41:32.736624939 +0000 UTC m=+99.059253308"
Sep 11 14:41:32.774640 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.774599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:32.786092 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.786046    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d9649dc55-sl4ns" podStartSLOduration=1.736046466 podStartE2EDuration="5.786029781s" totalImagesPullingTime="4.049983315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:32.785084913 +0000 UTC m=+99.107713284" watchObservedRunningTime="2026-09-11 14:41:32.786029781 +0000 UTC m=+99.108658148"
Sep 11 14:41:32.877072 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.876977    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:32.923858 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.923820    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft"]
Sep 11 14:41:32.927478 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.927433    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft"
Sep 11 14:41:32.930287 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.930257    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 14:41:32.930434 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.930326    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-b24s5\""
Sep 11 14:41:32.937538 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.937514    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft"]
Sep 11 14:41:32.976573 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:32.976539    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:33.077519 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:33.077449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0b1f5560-3141-4801-98b6-52578799f5ed-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qf6ft\" (UID: \"0b1f5560-3141-4801-98b6-52578799f5ed\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft"
Sep 11 14:41:33.178996 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:33.178917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0b1f5560-3141-4801-98b6-52578799f5ed-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qf6ft\" (UID: \"0b1f5560-3141-4801-98b6-52578799f5ed\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft"
Sep 11 14:41:33.179145 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:33.179052    2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 11 14:41:33.179324 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:33.179310    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1f5560-3141-4801-98b6-52578799f5ed-tls-certificates podName:0b1f5560-3141-4801-98b6-52578799f5ed nodeName:}" failed. No retries permitted until 2026-09-11 14:41:33.679290188 +0000 UTC m=+100.001918536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/0b1f5560-3141-4801-98b6-52578799f5ed-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-qf6ft" (UID: "0b1f5560-3141-4801-98b6-52578799f5ed") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 11 14:41:33.211289 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:33.211263    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:33.312327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:33.312289    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:33.373860 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:33.373823    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:33.656999 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:33.656965    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:33.683309 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:33.683282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0b1f5560-3141-4801-98b6-52578799f5ed-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qf6ft\" (UID: \"0b1f5560-3141-4801-98b6-52578799f5ed\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft"
Sep 11 14:41:33.685576 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:33.685545    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0b1f5560-3141-4801-98b6-52578799f5ed-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qf6ft\" (UID: \"0b1f5560-3141-4801-98b6-52578799f5ed\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft"
Sep 11 14:41:33.839015 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:33.838986    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft"
Sep 11 14:41:34.618787 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:34.618694    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft"]
Sep 11 14:41:34.621418 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:34.621387    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1f5560_3141_4801_98b6_52578799f5ed.slice/crio-7f74036994e0b9796ee65209bba4903858feb7976936765b47606d342cb42ac5 WatchSource:0}: Error finding container 7f74036994e0b9796ee65209bba4903858feb7976936765b47606d342cb42ac5: Status 404 returned error can't find the container with id 7f74036994e0b9796ee65209bba4903858feb7976936765b47606d342cb42ac5
Sep 11 14:41:34.686478 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:34.686424    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft" event={"ID":"0b1f5560-3141-4801-98b6-52578799f5ed","Type":"ContainerStarted","Data":"7f74036994e0b9796ee65209bba4903858feb7976936765b47606d342cb42ac5"}
Sep 11 14:41:34.688370 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:34.688343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7klw" event={"ID":"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c","Type":"ContainerStarted","Data":"80e569c10de0664e25e244871869634574ec60fe443c4bfa0f8a5a66a6c0e2e8"}
Sep 11 14:41:34.688481 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:34.688388    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7klw" event={"ID":"ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c","Type":"ContainerStarted","Data":"84b4727f77ff71c07a5461403f398ad5f874129ff50a8c1c1d0f2a13d8361f2f"}
Sep 11 14:41:34.710207 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:34.710163    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w7klw" podStartSLOduration=65.896797133 podStartE2EDuration="1m7.71015045s" totalImagesPullingTime="1.813353317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:34.708553686 +0000 UTC m=+101.031182046" watchObservedRunningTime="2026-09-11 14:41:34.71015045 +0000 UTC m=+101.032778820"
Sep 11 14:41:35.037336 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:35.037290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:35.692813 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:35.692778    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w7klw"
Sep 11 14:41:36.576471 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:36.576412    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9l6tg"
Sep 11 14:41:36.698407 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:36.698315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798c4c6c45-zl7zn" event={"ID":"4d2deea2-5322-4201-b2d4-08c50d4fb76e","Type":"ContainerStarted","Data":"b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0"}
Sep 11 14:41:36.699270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:36.699244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:36.721784 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:36.721736    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-798c4c6c45-zl7zn" podStartSLOduration=1.946609741 podStartE2EDuration="5.721717448s" totalImagesPullingTime="3.775107707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:36.72076716 +0000 UTC m=+103.043395531" watchObservedRunningTime="2026-09-11 14:41:36.721717448 +0000 UTC m=+103.044345818"
Sep 11 14:41:36.997273 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:36.997236    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:37.055157 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:37.055126    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:37.534370 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:37.534333    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:37.593932 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:37.593899    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:37.703730 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:37.703651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft" event={"ID":"0b1f5560-3141-4801-98b6-52578799f5ed","Type":"ContainerStarted","Data":"b90023f28e88e25afd396c68ddad91548bdff76e1c10fb0f3318b75bd529d0c8"}
Sep 11 14:41:37.704133 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:37.703888    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft"
Sep 11 14:41:37.708636 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:37.708614    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft"
Sep 11 14:41:37.721124 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:37.721083    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qf6ft" podStartSLOduration=2.965658193 podStartE2EDuration="5.721068505s" totalImagesPullingTime="2.755410312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:37.719965624 +0000 UTC m=+104.042593995" watchObservedRunningTime="2026-09-11 14:41:37.721068505 +0000 UTC m=+104.043696875"
Sep 11 14:41:37.938089 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:37.938051    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:38.009471 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:38.009434    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:38.270745 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:38.270660    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:38.596042 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:38.595947    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:39.190355 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.190323    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-659999cd6f-srzl2"]
Sep 11 14:41:39.192605 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.192587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.202006 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.201980    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 14:41:39.204197 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.204175    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-659999cd6f-srzl2"]
Sep 11 14:41:39.330682 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.330644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-serving-cert\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.330861 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.330787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbdzz\" (UniqueName: \"kubernetes.io/projected/b8943c90-72df-4971-8da6-d288d1f58f14-kube-api-access-cbdzz\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.330861 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.330840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-service-ca\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.331020 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.330988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-oauth-config\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.331126 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.331032    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-oauth-serving-cert\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.331126 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.331058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-console-config\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.331126 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.331091    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-trusted-ca-bundle\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.431571 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.431543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-serving-cert\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.431571 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.431579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbdzz\" (UniqueName: \"kubernetes.io/projected/b8943c90-72df-4971-8da6-d288d1f58f14-kube-api-access-cbdzz\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.431825 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.431602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-service-ca\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.431825 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.431755    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-oauth-config\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.431825 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.431782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-oauth-serving-cert\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.431825 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.431802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-console-config\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.432020 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.431834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-trusted-ca-bundle\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.432429 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.432394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-service-ca\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.432551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.432534    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-oauth-serving-cert\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.432667 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.432643    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-console-config\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.432722 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.432711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-trusted-ca-bundle\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.434078 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.434053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-serving-cert\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.434078 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.434056    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-oauth-config\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.440804 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.440750    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbdzz\" (UniqueName: \"kubernetes.io/projected/b8943c90-72df-4971-8da6-d288d1f58f14-kube-api-access-cbdzz\") pod \"console-659999cd6f-srzl2\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") " pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.501659 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.501624    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:39.774641 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:39.774610    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-659999cd6f-srzl2"]
Sep 11 14:41:39.777715 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:39.777686    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8943c90_72df_4971_8da6_d288d1f58f14.slice/crio-70d2ca3d05a120738d24eba7174351361f4faa334e4a8cf9b01c20a7447988f8 WatchSource:0}: Error finding container 70d2ca3d05a120738d24eba7174351361f4faa334e4a8cf9b01c20a7447988f8: Status 404 returned error can't find the container with id 70d2ca3d05a120738d24eba7174351361f4faa334e4a8cf9b01c20a7447988f8
Sep 11 14:41:40.715742 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:40.715708    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-659999cd6f-srzl2" event={"ID":"b8943c90-72df-4971-8da6-d288d1f58f14","Type":"ContainerStarted","Data":"768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e"}
Sep 11 14:41:40.716118 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:40.715756    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-659999cd6f-srzl2" event={"ID":"b8943c90-72df-4971-8da6-d288d1f58f14","Type":"ContainerStarted","Data":"70d2ca3d05a120738d24eba7174351361f4faa334e4a8cf9b01c20a7447988f8"}
Sep 11 14:41:40.741790 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:40.741740    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-659999cd6f-srzl2" podStartSLOduration=1.741726197 podStartE2EDuration="1.741726197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:40.740693252 +0000 UTC m=+107.063321623" watchObservedRunningTime="2026-09-11 14:41:40.741726197 +0000 UTC m=+107.064354635"
Sep 11 14:41:41.404000 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:41.403963    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:41.504253 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:41.504216    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:41.867540 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:41.867485    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:41.867911 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:41.867582    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:41.872311 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:41.872289    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:42.727251 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:42.727216    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:41:45.222724 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:45.222691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:45.314077 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:45.314041    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:45.700863 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:45.700781    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w7klw"
Sep 11 14:41:46.205376 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:46.205339    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:46.274036 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:46.274001    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:46.661532 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:46.661492    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:46.764146 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:46.764109    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:47.047284 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:47.047252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:47.975649 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:47.975593    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:48.125201 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:48.125168    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:48.204297 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:48.204259    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:49.108839 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.108801    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:49.299345 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.299296    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:49.380293 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.380211    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"]
Sep 11 14:41:49.383540 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.383522    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.390393 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.390365    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 14:41:49.390393 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.390381    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 14:41:49.390875 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.390729    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 14:41:49.394110 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.394094    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-l6kpg\""
Sep 11 14:41:49.395472 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.395434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wwpr\" (UniqueName: \"kubernetes.io/projected/574d7d83-4358-4a87-9500-0cc09631452f-kube-api-access-8wwpr\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.395550 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.395486    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/574d7d83-4358-4a87-9500-0cc09631452f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.395550 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.395537    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/574d7d83-4358-4a87-9500-0cc09631452f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.395668 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.395595    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/574d7d83-4358-4a87-9500-0cc09631452f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.415629 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.415596    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-666b8"]
Sep 11 14:41:49.418957 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.418930    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"]
Sep 11 14:41:49.419067 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.419023    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.419821 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.419792    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:49.422215 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.422194    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"]
Sep 11 14:41:49.422331 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.422284    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.423599 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.423580    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 14:41:49.424395 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.424379    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 14:41:49.424637 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.424614    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9pcp2\""
Sep 11 14:41:49.424758 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.424660    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 14:41:49.424883 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.424868    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 14:41:49.425200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.425184    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jvjqg\""
Sep 11 14:41:49.425793 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.425770    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 14:41:49.427380 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.427363    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 14:41:49.444469 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.444431    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"]
Sep 11 14:41:49.496615 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.496583    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-accelerators-collector-config\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.496615 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.496618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d344e80-b4d3-4ac9-9454-f1df618ba028-root\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.496808 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.496643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/574d7d83-4358-4a87-9500-0cc09631452f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.496808 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.496679    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d344e80-b4d3-4ac9-9454-f1df618ba028-metrics-client-ca\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.496808 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.496715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d344e80-b4d3-4ac9-9454-f1df618ba028-sys\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.496808 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:49.496721    2559 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 14:41:49.496808 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.496738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-textfile\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.496808 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:49.496783    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/574d7d83-4358-4a87-9500-0cc09631452f-openshift-state-metrics-tls podName:574d7d83-4358-4a87-9500-0cc09631452f nodeName:}" failed. No retries permitted until 2026-09-11 14:41:49.996760694 +0000 UTC m=+116.319389043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/574d7d83-4358-4a87-9500-0cc09631452f-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-vksz9" (UID: "574d7d83-4358-4a87-9500-0cc09631452f") : secret "openshift-state-metrics-tls" not found
Sep 11 14:41:49.496808 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.496800    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-wtmp\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.497065 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.496899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjczc\" (UniqueName: \"kubernetes.io/projected/4d344e80-b4d3-4ac9-9454-f1df618ba028-kube-api-access-jjczc\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.497065 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.496952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wwpr\" (UniqueName: \"kubernetes.io/projected/574d7d83-4358-4a87-9500-0cc09631452f-kube-api-access-8wwpr\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.497065 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.496973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/574d7d83-4358-4a87-9500-0cc09631452f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.497065 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.496992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.497065 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.497016    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/574d7d83-4358-4a87-9500-0cc09631452f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.497065 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.497035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-tls\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.497688 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.497664    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/574d7d83-4358-4a87-9500-0cc09631452f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.499295 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.499270    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/574d7d83-4358-4a87-9500-0cc09631452f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.502076 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.502057    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:49.502168 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.502099    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:49.507123 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.507105    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:49.516372 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.516350    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wwpr\" (UniqueName: \"kubernetes.io/projected/574d7d83-4358-4a87-9500-0cc09631452f-kube-api-access-8wwpr\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:49.570907 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.570862    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:49.598120 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598091    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d344e80-b4d3-4ac9-9454-f1df618ba028-root\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.598308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fd829058-20c0-48f3-b69e-d609e69ad9a9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.598308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598182    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d344e80-b4d3-4ac9-9454-f1df618ba028-metrics-client-ca\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.598308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598197    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d344e80-b4d3-4ac9-9454-f1df618ba028-root\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.598308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d344e80-b4d3-4ac9-9454-f1df618ba028-sys\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.598308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-textfile\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.598308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-wtmp\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.598308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjczc\" (UniqueName: \"kubernetes.io/projected/4d344e80-b4d3-4ac9-9454-f1df618ba028-kube-api-access-jjczc\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.598696 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598376    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d344e80-b4d3-4ac9-9454-f1df618ba028-sys\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.598696 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd829058-20c0-48f3-b69e-d609e69ad9a9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.598696 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-wtmp\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.598696 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.598696 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598520    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdvc\" (UniqueName: \"kubernetes.io/projected/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-api-access-bkdvc\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.598696 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598637    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-textfile\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.598696 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.599002 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.599002 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598769    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.599002 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598895    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d344e80-b4d3-4ac9-9454-f1df618ba028-metrics-client-ca\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.599002 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.598979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-tls\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.599195 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.599051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-accelerators-collector-config\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.599195 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:49.599080    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 14:41:49.599291 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:49.599203    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-tls podName:4d344e80-b4d3-4ac9-9454-f1df618ba028 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:50.099182746 +0000 UTC m=+116.421811100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-tls") pod "node-exporter-666b8" (UID: "4d344e80-b4d3-4ac9-9454-f1df618ba028") : secret "node-exporter-tls" not found
Sep 11 14:41:49.599539 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.599522    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-accelerators-collector-config\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.600886 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.600866    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.615442 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.615410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjczc\" (UniqueName: \"kubernetes.io/projected/4d344e80-b4d3-4ac9-9454-f1df618ba028-kube-api-access-jjczc\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:49.700295 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.700205    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fd829058-20c0-48f3-b69e-d609e69ad9a9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.700295 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.700278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd829058-20c0-48f3-b69e-d609e69ad9a9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.700551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.700300    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.700551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.700321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdvc\" (UniqueName: \"kubernetes.io/projected/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-api-access-bkdvc\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.700551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.700368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.700551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.700399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.700785 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.700731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fd829058-20c0-48f3-b69e-d609e69ad9a9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.700785 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:49.700759    2559 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 11 14:41:49.700890 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:49.700841    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-tls podName:fd829058-20c0-48f3-b69e-d609e69ad9a9 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:50.200819673 +0000 UTC m=+116.523448036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-v2f5g" (UID: "fd829058-20c0-48f3-b69e-d609e69ad9a9") : secret "kube-state-metrics-tls" not found
Sep 11 14:41:49.701113 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.701080    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.701205 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.701171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd829058-20c0-48f3-b69e-d609e69ad9a9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.702987 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.702965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.716173 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.716139    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdvc\" (UniqueName: \"kubernetes.io/projected/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-api-access-bkdvc\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:49.750929 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.750903    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:41:49.817518 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.817482    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:49.850209 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.850170    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-798c4c6c45-zl7zn"]
Sep 11 14:41:49.924841 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:49.924803    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:50.003082 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.003044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/574d7d83-4358-4a87-9500-0cc09631452f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:50.005392 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.005367    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/574d7d83-4358-4a87-9500-0cc09631452f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vksz9\" (UID: \"574d7d83-4358-4a87-9500-0cc09631452f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:50.012860 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.012835    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:50.103686 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.103649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-tls\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:50.103841 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:50.103812    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 14:41:50.103894 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:41:50.103884    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-tls podName:4d344e80-b4d3-4ac9-9454-f1df618ba028 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:51.103865231 +0000 UTC m=+117.426493595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-tls") pod "node-exporter-666b8" (UID: "4d344e80-b4d3-4ac9-9454-f1df618ba028") : secret "node-exporter-tls" not found
Sep 11 14:41:50.118157 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.118128    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:50.158367 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.158334    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b7f94f844-q7kdx"]
Sep 11 14:41:50.162704 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.162675    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:41:50.204159 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.204122    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:50.206389 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.206368    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd829058-20c0-48f3-b69e-d609e69ad9a9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v2f5g\" (UID: \"fd829058-20c0-48f3-b69e-d609e69ad9a9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:50.218046 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.218020    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:50.293274 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.293194    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"
Sep 11 14:41:50.323315 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.323285    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:50.334917 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.334895    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"
Sep 11 14:41:50.417819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.417785    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:50.523335 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.523304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:50.573221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.573190    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9"]
Sep 11 14:41:50.578267 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:50.578243    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574d7d83_4358_4a87_9500_0cc09631452f.slice/crio-efdc60d1ead1bfc0e8e7c551140defbf7f5a292db434aa0368d37b63a7804af2 WatchSource:0}: Error finding container efdc60d1ead1bfc0e8e7c551140defbf7f5a292db434aa0368d37b63a7804af2: Status 404 returned error can't find the container with id efdc60d1ead1bfc0e8e7c551140defbf7f5a292db434aa0368d37b63a7804af2
Sep 11 14:41:50.622212 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.622193    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g"]
Sep 11 14:41:50.623957 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:50.623928    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd829058_20c0_48f3_b69e_d609e69ad9a9.slice/crio-bb548b603cdd41e3bbf3137d5e65623081c3feed31df61c01516b74cd5cdae69 WatchSource:0}: Error finding container bb548b603cdd41e3bbf3137d5e65623081c3feed31df61c01516b74cd5cdae69: Status 404 returned error can't find the container with id bb548b603cdd41e3bbf3137d5e65623081c3feed31df61c01516b74cd5cdae69
Sep 11 14:41:50.625523 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.625498    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:50.752211 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.752178    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g" event={"ID":"fd829058-20c0-48f3-b69e-d609e69ad9a9","Type":"ContainerStarted","Data":"bb548b603cdd41e3bbf3137d5e65623081c3feed31df61c01516b74cd5cdae69"}
Sep 11 14:41:50.754100 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.754076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9" event={"ID":"574d7d83-4358-4a87-9500-0cc09631452f","Type":"ContainerStarted","Data":"0fd452636d5cc2f4b168d9938364c4ebcf396e11c42803d3aa75f7ae6773e571"}
Sep 11 14:41:50.754231 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.754123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9" event={"ID":"574d7d83-4358-4a87-9500-0cc09631452f","Type":"ContainerStarted","Data":"45ebf0fa2c1b23fe72795acc28c397a510cbb40e11aa839610a4288f51df6bff"}
Sep 11 14:41:50.754231 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.754142    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9" event={"ID":"574d7d83-4358-4a87-9500-0cc09631452f","Type":"ContainerStarted","Data":"efdc60d1ead1bfc0e8e7c551140defbf7f5a292db434aa0368d37b63a7804af2"}
Sep 11 14:41:50.829552 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:50.829474    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:51.112825 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:51.112744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-tls\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:51.115397 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:51.115370    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d344e80-b4d3-4ac9-9454-f1df618ba028-node-exporter-tls\") pod \"node-exporter-666b8\" (UID: \"4d344e80-b4d3-4ac9-9454-f1df618ba028\") " pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:51.230076 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:51.230046    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-666b8"
Sep 11 14:41:51.241567 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:51.241543    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d344e80_b4d3_4ac9_9454_f1df618ba028.slice/crio-b827376e82918b5dbd559b8f87aa47b1bd8fb20eaef51391fc0f3e94b81dc570 WatchSource:0}: Error finding container b827376e82918b5dbd559b8f87aa47b1bd8fb20eaef51391fc0f3e94b81dc570: Status 404 returned error can't find the container with id b827376e82918b5dbd559b8f87aa47b1bd8fb20eaef51391fc0f3e94b81dc570
Sep 11 14:41:51.762592 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:51.762559    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666b8" event={"ID":"4d344e80-b4d3-4ac9-9454-f1df618ba028","Type":"ContainerStarted","Data":"b827376e82918b5dbd559b8f87aa47b1bd8fb20eaef51391fc0f3e94b81dc570"}
Sep 11 14:41:52.362003 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.361932    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:52.431415 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.431365    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:52.508860 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.508816    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:52.768445 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.768417    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g" event={"ID":"fd829058-20c0-48f3-b69e-d609e69ad9a9","Type":"ContainerStarted","Data":"9935287614e2e994b7c6312385fc673286788dabcfd0c75ed9a4ed66d4cf6054"}
Sep 11 14:41:52.768568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.768514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g" event={"ID":"fd829058-20c0-48f3-b69e-d609e69ad9a9","Type":"ContainerStarted","Data":"9849d1d05c257f75ef5e7fd3a40a674b42bb7ef76a790b784b6d3aa6255a12b4"}
Sep 11 14:41:52.768568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.768550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g" event={"ID":"fd829058-20c0-48f3-b69e-d609e69ad9a9","Type":"ContainerStarted","Data":"8f40b2709278c9fca1812cf1300287dcc46ffcaa23a4c01049efe7312782cec5"}
Sep 11 14:41:52.770523 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.770495    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9" event={"ID":"574d7d83-4358-4a87-9500-0cc09631452f","Type":"ContainerStarted","Data":"32e56b2b76b9584199cb283a1de5463083ff742345a1f41b52c436fc84c4eca2"}
Sep 11 14:41:52.772070 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.772048    2559 generic.go:356] "Generic (PLEG): container finished" podID="4d344e80-b4d3-4ac9-9454-f1df618ba028" containerID="685e3be99f1ec8457b27f8f3009909982a65908bb8979c7ea0cea7897535f87d" exitCode=0
Sep 11 14:41:52.772184 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.772086    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666b8" event={"ID":"4d344e80-b4d3-4ac9-9454-f1df618ba028","Type":"ContainerDied","Data":"685e3be99f1ec8457b27f8f3009909982a65908bb8979c7ea0cea7897535f87d"}
Sep 11 14:41:52.794861 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.794818    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v2f5g" podStartSLOduration=2.384448179 podStartE2EDuration="3.794805024s" totalImagesPullingTime="1.410356845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:52.793114536 +0000 UTC m=+119.115742919" watchObservedRunningTime="2026-09-11 14:41:52.794805024 +0000 UTC m=+119.117433394"
Sep 11 14:41:52.813556 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.813513    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vksz9" podStartSLOduration=2.475239876 podStartE2EDuration="3.813498635s" totalImagesPullingTime="1.338258759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:52.812844338 +0000 UTC m=+119.135472708" watchObservedRunningTime="2026-09-11 14:41:52.813498635 +0000 UTC m=+119.136127005"
Sep 11 14:41:52.990229 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:52.990191    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:53.778403 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:53.778370    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666b8" event={"ID":"4d344e80-b4d3-4ac9-9454-f1df618ba028","Type":"ContainerStarted","Data":"6feb4fca03ff0d1a3755e8ab02ccfde8d2ea216c59a30cf9d18d2a9179671c02"}
Sep 11 14:41:53.778880 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:53.778481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666b8" event={"ID":"4d344e80-b4d3-4ac9-9454-f1df618ba028","Type":"ContainerStarted","Data":"98ae79267ad3f4abcaca2775478b8476d48fc26d24a7be22dffcd876b12681a2"}
Sep 11 14:41:53.812172 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:53.812134    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-666b8" podStartSLOduration=3.365387988 podStartE2EDuration="4.812116876s" totalImagesPullingTime="1.382681212s" totalInitContainerRuntime="64.047676ms" isStatefulPod=true podCreationTimestamp="2026-09-11 14:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:53.810520399 +0000 UTC m=+120.133148769" watchObservedRunningTime="2026-09-11 14:41:53.812116876 +0000 UTC m=+120.134745245"
Sep 11 14:41:54.164231 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.164135    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:54.246319 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.246289    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:54.360017 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.359983    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:54.460950 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.460858    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:54.684307 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.684273    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-74849c8dd-gb62q"]
Sep 11 14:41:54.688679 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.688662    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.691567 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.691535    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 14:41:54.691694 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.691604    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 14:41:54.691694 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.691623    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 14:41:54.691694 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.691624    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 14:41:54.691694 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.691688    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bcfcc\""
Sep 11 14:41:54.691853 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.691624    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 14:41:54.696662 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.696642    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 14:41:54.702127 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.702107    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74849c8dd-gb62q"]
Sep 11 14:41:54.744597 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.744574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.744700 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.744627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/786b6811-09b6-4f34-93b3-a545605c0817-metrics-client-ca\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.744700 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.744680    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-telemeter-client-tls\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.744700 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.744698    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-federate-client-tls\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.744826 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.744728    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-secret-telemeter-client\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.744826 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.744774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b6811-09b6-4f34-93b3-a545605c0817-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.744826 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.744805    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b6811-09b6-4f34-93b3-a545605c0817-serving-certs-ca-bundle\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.744826 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.744825    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z5wf\" (UniqueName: \"kubernetes.io/projected/786b6811-09b6-4f34-93b3-a545605c0817-kube-api-access-2z5wf\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.845486 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.845433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b6811-09b6-4f34-93b3-a545605c0817-serving-certs-ca-bundle\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.845486 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.845487    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z5wf\" (UniqueName: \"kubernetes.io/projected/786b6811-09b6-4f34-93b3-a545605c0817-kube-api-access-2z5wf\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.846007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.845519    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.846007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.845572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/786b6811-09b6-4f34-93b3-a545605c0817-metrics-client-ca\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.846007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.845639    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-telemeter-client-tls\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.846007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.845677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-federate-client-tls\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.846007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.845718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-secret-telemeter-client\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.846007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.845914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b6811-09b6-4f34-93b3-a545605c0817-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.846400 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.846376    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/786b6811-09b6-4f34-93b3-a545605c0817-metrics-client-ca\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.846501 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.846448    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b6811-09b6-4f34-93b3-a545605c0817-serving-certs-ca-bundle\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.846807 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.846783    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b6811-09b6-4f34-93b3-a545605c0817-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.848319 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.848289    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.848406 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.848341    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-secret-telemeter-client\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.848625 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.848608    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-federate-client-tls\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.848693 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.848665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/786b6811-09b6-4f34-93b3-a545605c0817-telemeter-client-tls\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.853394 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.853367    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:54.854732 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.854710    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z5wf\" (UniqueName: \"kubernetes.io/projected/786b6811-09b6-4f34-93b3-a545605c0817-kube-api-access-2z5wf\") pod \"telemeter-client-74849c8dd-gb62q\" (UID: \"786b6811-09b6-4f34-93b3-a545605c0817\") " pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:54.980662 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.980626    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:54.998581 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:54.998532    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q"
Sep 11 14:41:55.302035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.301854    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74849c8dd-gb62q"]
Sep 11 14:41:55.644539 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.644441    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 14:41:55.645928 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.645887    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:41:55.649700 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.649679    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.653930 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.653884    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 14:41:55.654026 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.653961    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 14:41:55.654026 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.654006    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 14:41:55.654155 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.654109    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 14:41:55.654155 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.654117    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 14:41:55.654578 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.654562    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 14:41:55.655124 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.655106    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 14:41:55.655211 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.655170    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 14:41:55.655274 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.655217    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-95esp3gqs3sam\""
Sep 11 14:41:55.655274 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.655227    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 14:41:55.655274 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.655233    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 14:41:55.655489 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.655449    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 14:41:55.655615 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.655596    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 14:41:55.655743 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.655721    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 14:41:55.655860 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.655844    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8vm88\""
Sep 11 14:41:55.657549 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.657533    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 14:41:55.659958 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.659942    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 14:41:55.674597 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.674579    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 14:41:55.751550 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.751671 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751555    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.751671 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751587    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.751671 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.751775 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751681    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe9ae9f8-afe3-43ee-855e-ca4181316d21-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.751775 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751718    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.751775 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.751870 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.751870 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.751870 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751810    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-config\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.751870 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751827    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.751870 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlwcb\" (UniqueName: \"kubernetes.io/projected/fe9ae9f8-afe3-43ee-855e-ca4181316d21-kube-api-access-qlwcb\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.752079 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.752079 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751906    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe9ae9f8-afe3-43ee-855e-ca4181316d21-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.752079 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.752079 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.751949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.752079 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.752017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.752079 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.752055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.752301 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.752092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.752301 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.752118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.786484 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.786448    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q" event={"ID":"786b6811-09b6-4f34-93b3-a545605c0817","Type":"ContainerStarted","Data":"2d83e8d07b283c19be7b885a463209ec7e23dcc01e8385b40bd68552622a0d6c"}
Sep 11 14:41:55.853381 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-config\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlwcb\" (UniqueName: \"kubernetes.io/projected/fe9ae9f8-afe3-43ee-855e-ca4181316d21-kube-api-access-qlwcb\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853486    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe9ae9f8-afe3-43ee-855e-ca4181316d21-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853532    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853749    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.853792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.854388 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.854388 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853883    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.854388 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.854388 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe9ae9f8-afe3-43ee-855e-ca4181316d21-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.854388 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.853978    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.854388 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.854008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.854388 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.854034    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.854388 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.854318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.854829 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.854641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.855735 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.855713    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.856537 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.856514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.857495 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.856518    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.857495 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.857159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.857495 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.857181    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.857495 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.857187    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe9ae9f8-afe3-43ee-855e-ca4181316d21-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.857495 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.857296    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.857495 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.857380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.858101 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.858055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-config\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.859221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.859128    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.859316 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.859218    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.864660 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.859507    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.864660 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.859580    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe9ae9f8-afe3-43ee-855e-ca4181316d21-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.864660 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.859721    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.864660 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.861309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.864660 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.863161    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlwcb\" (UniqueName: \"kubernetes.io/projected/fe9ae9f8-afe3-43ee-855e-ca4181316d21-kube-api-access-qlwcb\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.865169 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.865148    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe9ae9f8-afe3-43ee-855e-ca4181316d21-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.865600 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.865579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe9ae9f8-afe3-43ee-855e-ca4181316d21-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe9ae9f8-afe3-43ee-855e-ca4181316d21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:55.960890 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:55.960821    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:41:56.251661 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:56.251630    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 14:41:56.255144 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:56.255113    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9ae9f8_afe3_43ee_855e_ca4181316d21.slice/crio-03ee85618a526b1ed3c358a9538d3c405ff3e3907101fa4d7aa655bd4a75f28a WatchSource:0}: Error finding container 03ee85618a526b1ed3c358a9538d3c405ff3e3907101fa4d7aa655bd4a75f28a: Status 404 returned error can't find the container with id 03ee85618a526b1ed3c358a9538d3c405ff3e3907101fa4d7aa655bd4a75f28a
Sep 11 14:41:56.793446 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:56.793334    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe9ae9f8-afe3-43ee-855e-ca4181316d21","Type":"ContainerStarted","Data":"03ee85618a526b1ed3c358a9538d3c405ff3e3907101fa4d7aa655bd4a75f28a"}
Sep 11 14:41:57.799368 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:57.799285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q" event={"ID":"786b6811-09b6-4f34-93b3-a545605c0817","Type":"ContainerStarted","Data":"05595ee067d7447f94bea63a915362888986a3da37187b11620916be9b69a851"}
Sep 11 14:41:57.799368 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:57.799355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q" event={"ID":"786b6811-09b6-4f34-93b3-a545605c0817","Type":"ContainerStarted","Data":"d460bda7ee428bad410da971883f4a9a8d930ab64c054b19a87cc81d1e78eed2"}
Sep 11 14:41:57.799828 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:57.799382    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q" event={"ID":"786b6811-09b6-4f34-93b3-a545605c0817","Type":"ContainerStarted","Data":"d67742c7ce839a5e6c172a678637a17eb827f20766d6116274e9498540a9f5ee"}
Sep 11 14:41:57.800891 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:57.800868    2559 generic.go:356] "Generic (PLEG): container finished" podID="fe9ae9f8-afe3-43ee-855e-ca4181316d21" containerID="8f8fe3e670f5fd5460c0713953622b950ace8d265fc05b09020d65111e7dca45" exitCode=0
Sep 11 14:41:57.801007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:57.800928    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe9ae9f8-afe3-43ee-855e-ca4181316d21","Type":"ContainerDied","Data":"8f8fe3e670f5fd5460c0713953622b950ace8d265fc05b09020d65111e7dca45"}
Sep 11 14:41:57.824840 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:57.824785    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-74849c8dd-gb62q" podStartSLOduration=1.634669445 podStartE2EDuration="3.824770527s" totalImagesPullingTime="2.190101082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:57.822525015 +0000 UTC m=+124.145153385" watchObservedRunningTime="2026-09-11 14:41:57.824770527 +0000 UTC m=+124.147398896"
Sep 11 14:41:57.988591 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:57.988552    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65575bf675-4grsz"]
Sep 11 14:41:57.994353 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:57.994329    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.002467 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.002431    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65575bf675-4grsz"]
Sep 11 14:41:58.076297 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.076222    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-trusted-ca-bundle\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.076297 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.076258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-console-config\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.076297 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.076286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-oauth-config\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.076512 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.076423    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-service-ca\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.076512 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.076449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-964rg\" (UniqueName: \"kubernetes.io/projected/d9057648-04d8-4caa-8dcc-085f2df0431c-kube-api-access-964rg\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.076578 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.076522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-serving-cert\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.076578 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.076538    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-oauth-serving-cert\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.177191 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.177163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-serving-cert\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.177191 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.177194    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-oauth-serving-cert\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.177398 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.177217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-trusted-ca-bundle\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.177398 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.177249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-console-config\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.177398 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.177273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-oauth-config\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.177398 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.177346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-service-ca\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.177398 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.177376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-964rg\" (UniqueName: \"kubernetes.io/projected/d9057648-04d8-4caa-8dcc-085f2df0431c-kube-api-access-964rg\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.178137 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.178111    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-oauth-serving-cert\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.178360 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.178148    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-console-config\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.178360 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.178208    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-service-ca\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.178573 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.178389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-trusted-ca-bundle\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.179890 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.179869    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-oauth-config\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.179969 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.179955    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-serving-cert\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.186336 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.186319    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-964rg\" (UniqueName: \"kubernetes.io/projected/d9057648-04d8-4caa-8dcc-085f2df0431c-kube-api-access-964rg\") pod \"console-65575bf675-4grsz\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") " pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.304159 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.304124    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:41:58.597510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.597412    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65575bf675-4grsz"]
Sep 11 14:41:58.600246 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:41:58.600216    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9057648_04d8_4caa_8dcc_085f2df0431c.slice/crio-1f3e9280124356f1c6257409c4f3e6093fe23fd82f85240e327e81ada9d09d9b WatchSource:0}: Error finding container 1f3e9280124356f1c6257409c4f3e6093fe23fd82f85240e327e81ada9d09d9b: Status 404 returned error can't find the container with id 1f3e9280124356f1c6257409c4f3e6093fe23fd82f85240e327e81ada9d09d9b
Sep 11 14:41:58.809391 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.809331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65575bf675-4grsz" event={"ID":"d9057648-04d8-4caa-8dcc-085f2df0431c","Type":"ContainerStarted","Data":"9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790"}
Sep 11 14:41:58.809879 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.809449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65575bf675-4grsz" event={"ID":"d9057648-04d8-4caa-8dcc-085f2df0431c","Type":"ContainerStarted","Data":"1f3e9280124356f1c6257409c4f3e6093fe23fd82f85240e327e81ada9d09d9b"}
Sep 11 14:41:58.832822 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:41:58.832773    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65575bf675-4grsz" podStartSLOduration=1.832755377 podStartE2EDuration="1.832755377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:58.830904374 +0000 UTC m=+125.153532749" watchObservedRunningTime="2026-09-11 14:41:58.832755377 +0000 UTC m=+125.155383749"
Sep 11 14:42:00.170627 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:00.170590    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:42:00.258816 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:00.258777    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:42:01.826103 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:01.826066    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe9ae9f8-afe3-43ee-855e-ca4181316d21","Type":"ContainerStarted","Data":"88c9abc5e7b8bd1270fa28089712320df6a41c57585a10d465b9a2e1f052ed78"}
Sep 11 14:42:01.826504 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:01.826171    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe9ae9f8-afe3-43ee-855e-ca4181316d21","Type":"ContainerStarted","Data":"c90b3f61c64c88cf00bb25030bd61a7560ce7953a59bb866187b03913addf3fd"}
Sep 11 14:42:03.837342 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:03.837309    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe9ae9f8-afe3-43ee-855e-ca4181316d21","Type":"ContainerStarted","Data":"83ae5d3a9f41c8d0eb14f9c0e1f23b9a400b147bd1bd53f4548ddd108dd35faa"}
Sep 11 14:42:03.837758 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:03.837372    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe9ae9f8-afe3-43ee-855e-ca4181316d21","Type":"ContainerStarted","Data":"c787a599a3cbcc57406466d59de1c4101e03951ad4f21cfe3b918f1de7d094be"}
Sep 11 14:42:03.837758 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:03.837401    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe9ae9f8-afe3-43ee-855e-ca4181316d21","Type":"ContainerStarted","Data":"e465bafd76b65f653f6576e64db603cad1f13fa15be93d8edbe8cd4e0d1fb1e7"}
Sep 11 14:42:03.837758 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:03.837424    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe9ae9f8-afe3-43ee-855e-ca4181316d21","Type":"ContainerStarted","Data":"78ade597de5d2d9004a7c50f7efbea5cf3a3717719ba65f5c889a2a81426cd89"}
Sep 11 14:42:03.880986 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:03.880924    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.196040737 podStartE2EDuration="8.880905371s" totalImagesPullingTime="6.667986694s" totalInitContainerRuntime="16.87794ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:03.878932437 +0000 UTC m=+130.201560809" watchObservedRunningTime="2026-09-11 14:42:03.880905371 +0000 UTC m=+130.203533744"
Sep 11 14:42:04.035188 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:04.035159    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:42:04.037408 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:04.037377    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde006ea-2cba-4b23-b83e-6f7703193a31-metrics-certs\") pod \"network-metrics-daemon-2snh4\" (UID: \"fde006ea-2cba-4b23-b83e-6f7703193a31\") " pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:42:04.296247 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:04.296163    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jjd7d\""
Sep 11 14:42:04.303617 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:04.303589    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2snh4"
Sep 11 14:42:04.581210 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:04.581189    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2snh4"]
Sep 11 14:42:04.583216 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:42:04.583190    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde006ea_2cba_4b23_b83e_6f7703193a31.slice/crio-6145d60e527c1647227b033c1564406c80f9959fe123bb66c713e947bed96016 WatchSource:0}: Error finding container 6145d60e527c1647227b033c1564406c80f9959fe123bb66c713e947bed96016: Status 404 returned error can't find the container with id 6145d60e527c1647227b033c1564406c80f9959fe123bb66c713e947bed96016
Sep 11 14:42:04.842272 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:04.842187    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2snh4" event={"ID":"fde006ea-2cba-4b23-b83e-6f7703193a31","Type":"ContainerStarted","Data":"6145d60e527c1647227b033c1564406c80f9959fe123bb66c713e947bed96016"}
Sep 11 14:42:05.850602 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:05.850571    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2snh4" event={"ID":"fde006ea-2cba-4b23-b83e-6f7703193a31","Type":"ContainerStarted","Data":"8f96b086c2d51d34ee267ba4bc41a33517406d6bb82e5cdd00fd41955cb1b694"}
Sep 11 14:42:05.851044 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:05.850622    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2snh4" event={"ID":"fde006ea-2cba-4b23-b83e-6f7703193a31","Type":"ContainerStarted","Data":"bc20b8d01198ab9df6abb27cd707d32b3b755e4884cacff0ad882e5d67a2f81f"}
Sep 11 14:42:05.852143 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:05.852123    2559 generic.go:356] "Generic (PLEG): container finished" podID="9c38c07c-2f0c-4e2a-a730-1280542e9fdc" containerID="46f1a70084aafc9e48f3f08516f5a5b0c38cfc77782efc0d022441d11bbf94db" exitCode=0
Sep 11 14:42:05.852238 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:05.852170    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk" event={"ID":"9c38c07c-2f0c-4e2a-a730-1280542e9fdc","Type":"ContainerDied","Data":"46f1a70084aafc9e48f3f08516f5a5b0c38cfc77782efc0d022441d11bbf94db"}
Sep 11 14:42:05.852437 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:05.852425    2559 scope.go:117] "RemoveContainer" containerID="46f1a70084aafc9e48f3f08516f5a5b0c38cfc77782efc0d022441d11bbf94db"
Sep 11 14:42:05.879438 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:05.879375    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2snh4" podStartSLOduration=130.825334648 podStartE2EDuration="2m11.879355804s" totalImagesPullingTime="1.054021156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:05.8776462 +0000 UTC m=+132.200274570" watchObservedRunningTime="2026-09-11 14:42:05.879355804 +0000 UTC m=+132.201984176"
Sep 11 14:42:05.961997 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:05.961965    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:42:06.077137 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:06.077096    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:42:06.174021 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:06.173934    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:42:06.858007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:06.857967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-hkdtk" event={"ID":"9c38c07c-2f0c-4e2a-a730-1280542e9fdc","Type":"ContainerStarted","Data":"3e91e1017574627df6f8b8bb4b12d2209de3ed7da464ae8ea08a5b48698a7d2b"}
Sep 11 14:42:08.304774 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:08.304735    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:42:08.305157 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:08.304801    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:42:08.309481 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:08.309445    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:42:08.869495 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:08.869467    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:42:08.927156 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:08.927124    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-659999cd6f-srzl2"]
Sep 11 14:42:09.047008 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:09.046974    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:42:09.149923 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:09.149834    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:42:09.226474 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:09.226419    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:42:13.188298 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:13.188253    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:42:14.873879 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:14.873842    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-798c4c6c45-zl7zn" podUID="4d2deea2-5322-4201-b2d4-08c50d4fb76e" containerName="console" containerID="cri-o://b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0" gracePeriod=15
Sep 11 14:42:15.105413 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.105385    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:42:15.177199 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.177114    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx" podUID="b5e9fa73-5376-4267-a1d3-90625b5f0fd6" containerName="registry" containerID="cri-o://81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71" gracePeriod=30
Sep 11 14:42:15.238934 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.238903    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzsp5\" (UniqueName: \"kubernetes.io/projected/4d2deea2-5322-4201-b2d4-08c50d4fb76e-kube-api-access-fzsp5\") pod \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") "
Sep 11 14:42:15.239075 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.238982    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-service-ca\") pod \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") "
Sep 11 14:42:15.239075 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.239024    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-oauth-serving-cert\") pod \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") "
Sep 11 14:42:15.239075 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.239065    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-oauth-config\") pod \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") "
Sep 11 14:42:15.239241 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.239093    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-config\") pod \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") "
Sep 11 14:42:15.239241 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.239143    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-serving-cert\") pod \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\" (UID: \"4d2deea2-5322-4201-b2d4-08c50d4fb76e\") "
Sep 11 14:42:15.239384 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.239354    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-service-ca" (OuterVolumeSpecName: "service-ca") pod "4d2deea2-5322-4201-b2d4-08c50d4fb76e" (UID: "4d2deea2-5322-4201-b2d4-08c50d4fb76e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:42:15.239558 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.239521    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4d2deea2-5322-4201-b2d4-08c50d4fb76e" (UID: "4d2deea2-5322-4201-b2d4-08c50d4fb76e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:42:15.239558 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.239543    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-config" (OuterVolumeSpecName: "console-config") pod "4d2deea2-5322-4201-b2d4-08c50d4fb76e" (UID: "4d2deea2-5322-4201-b2d4-08c50d4fb76e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:42:15.241249 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.241217    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4d2deea2-5322-4201-b2d4-08c50d4fb76e" (UID: "4d2deea2-5322-4201-b2d4-08c50d4fb76e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:42:15.241338 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.241259    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4d2deea2-5322-4201-b2d4-08c50d4fb76e" (UID: "4d2deea2-5322-4201-b2d4-08c50d4fb76e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:42:15.241338 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.241271    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2deea2-5322-4201-b2d4-08c50d4fb76e-kube-api-access-fzsp5" (OuterVolumeSpecName: "kube-api-access-fzsp5") pod "4d2deea2-5322-4201-b2d4-08c50d4fb76e" (UID: "4d2deea2-5322-4201-b2d4-08c50d4fb76e"). InnerVolumeSpecName "kube-api-access-fzsp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:42:15.340381 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.340183    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzsp5\" (UniqueName: \"kubernetes.io/projected/4d2deea2-5322-4201-b2d4-08c50d4fb76e-kube-api-access-fzsp5\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.340381 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.340215    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.340381 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.340229    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-oauth-serving-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.340381 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.340243    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-oauth-config\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.340381 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.340258    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-config\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.340381 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.340271    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d2deea2-5322-4201-b2d4-08c50d4fb76e-console-serving-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.397141 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.397117    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:42:15.541927 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.541899    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-ca-trust-extracted\") pod \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") "
Sep 11 14:42:15.542101 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.541966    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-certificates\") pod \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") "
Sep 11 14:42:15.542101 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.541989    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-image-registry-private-configuration\") pod \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") "
Sep 11 14:42:15.542101 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.542028    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") pod \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") "
Sep 11 14:42:15.542268 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.542193    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-installation-pull-secrets\") pod \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") "
Sep 11 14:42:15.542317 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.542269    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbpq9\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-kube-api-access-lbpq9\") pod \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") "
Sep 11 14:42:15.542317 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.542299    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-trusted-ca\") pod \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") "
Sep 11 14:42:15.542422 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.542338    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-bound-sa-token\") pod \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\" (UID: \"b5e9fa73-5376-4267-a1d3-90625b5f0fd6\") "
Sep 11 14:42:15.542422 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.542393    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b5e9fa73-5376-4267-a1d3-90625b5f0fd6" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:42:15.542658 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.542635    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-certificates\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.542986 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.542951    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b5e9fa73-5376-4267-a1d3-90625b5f0fd6" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:42:15.544618 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.544588    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b5e9fa73-5376-4267-a1d3-90625b5f0fd6" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:42:15.544724 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.544622    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b5e9fa73-5376-4267-a1d3-90625b5f0fd6" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:42:15.544724 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.544649    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b5e9fa73-5376-4267-a1d3-90625b5f0fd6" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:42:15.544816 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.544736    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b5e9fa73-5376-4267-a1d3-90625b5f0fd6" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:42:15.544816 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.544757    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-kube-api-access-lbpq9" (OuterVolumeSpecName: "kube-api-access-lbpq9") pod "b5e9fa73-5376-4267-a1d3-90625b5f0fd6" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6"). InnerVolumeSpecName "kube-api-access-lbpq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:42:15.550349 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.550326    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b5e9fa73-5376-4267-a1d3-90625b5f0fd6" (UID: "b5e9fa73-5376-4267-a1d3-90625b5f0fd6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:42:15.643214 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.643180    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-ca-trust-extracted\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.643214 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.643209    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-image-registry-private-configuration\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.643214 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.643219    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-registry-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.643410 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.643230    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-installation-pull-secrets\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.643410 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.643239    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbpq9\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-kube-api-access-lbpq9\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.643410 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.643246    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-trusted-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.643410 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.643255    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5e9fa73-5376-4267-a1d3-90625b5f0fd6-bound-sa-token\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:15.906284 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.906246    2559 generic.go:356] "Generic (PLEG): container finished" podID="b5e9fa73-5376-4267-a1d3-90625b5f0fd6" containerID="81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71" exitCode=0
Sep 11 14:42:15.906698 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.906294    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx"
Sep 11 14:42:15.906698 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.906333    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx" event={"ID":"b5e9fa73-5376-4267-a1d3-90625b5f0fd6","Type":"ContainerDied","Data":"81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71"}
Sep 11 14:42:15.906698 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.906381    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7f94f844-q7kdx" event={"ID":"b5e9fa73-5376-4267-a1d3-90625b5f0fd6","Type":"ContainerDied","Data":"4a891215cfbf180d9ea539b4137f17d0c9c8904c83af0db8634b492d755f87b6"}
Sep 11 14:42:15.906698 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.906412    2559 scope.go:117] "RemoveContainer" containerID="81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71"
Sep 11 14:42:15.907831 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.907756    2559 generic.go:356] "Generic (PLEG): container finished" podID="4d2deea2-5322-4201-b2d4-08c50d4fb76e" containerID="b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0" exitCode=0
Sep 11 14:42:15.907831 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.907785    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798c4c6c45-zl7zn" event={"ID":"4d2deea2-5322-4201-b2d4-08c50d4fb76e","Type":"ContainerDied","Data":"b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0"}
Sep 11 14:42:15.907831 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.907804    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798c4c6c45-zl7zn"
Sep 11 14:42:15.907831 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.907830    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798c4c6c45-zl7zn" event={"ID":"4d2deea2-5322-4201-b2d4-08c50d4fb76e","Type":"ContainerDied","Data":"a9dec956d089bc0bb126473fb9005c5784264b76b8cff078356abb04d0333d3d"}
Sep 11 14:42:15.916352 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.916336    2559 scope.go:117] "RemoveContainer" containerID="81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71"
Sep 11 14:42:15.916636 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:42:15.916616    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71\": container with ID starting with 81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71 not found: ID does not exist" containerID="81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71"
Sep 11 14:42:15.916720 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.916646    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71"} err="failed to get container status \"81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71\": rpc error: code = NotFound desc = could not find container \"81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71\": container with ID starting with 81954a3a6980bceeb3452017d779939f27a5bd934f80953c38caef1747c32b71 not found: ID does not exist"
Sep 11 14:42:15.916720 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.916686    2559 scope.go:117] "RemoveContainer" containerID="b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0"
Sep 11 14:42:15.925695 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.925682    2559 scope.go:117] "RemoveContainer" containerID="b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0"
Sep 11 14:42:15.926086 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:42:15.926061    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0\": container with ID starting with b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0 not found: ID does not exist" containerID="b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0"
Sep 11 14:42:15.926184 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.926096    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0"} err="failed to get container status \"b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0\": rpc error: code = NotFound desc = could not find container \"b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0\": container with ID starting with b2a7281c5a66b619243011e7a13b6f027245ce37ae4f44f10294465fd83094b0 not found: ID does not exist"
Sep 11 14:42:15.930139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.930106    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b7f94f844-q7kdx"]
Sep 11 14:42:15.934405 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.934386    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b7f94f844-q7kdx"]
Sep 11 14:42:15.948913 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.948890    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-798c4c6c45-zl7zn"]
Sep 11 14:42:15.953238 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:15.953218    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-798c4c6c45-zl7zn"]
Sep 11 14:42:16.179140 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:16.179060    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d2deea2-5322-4201-b2d4-08c50d4fb76e" path="/var/lib/kubelet/pods/4d2deea2-5322-4201-b2d4-08c50d4fb76e/volumes"
Sep 11 14:42:16.179493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:16.179481    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5e9fa73-5376-4267-a1d3-90625b5f0fd6" path="/var/lib/kubelet/pods/b5e9fa73-5376-4267-a1d3-90625b5f0fd6/volumes"
Sep 11 14:42:20.931634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:20.931600    2559 generic.go:356] "Generic (PLEG): container finished" podID="b35287e9-9950-4b9d-af5c-969dd954fead" containerID="8e776083509e1e05204c4f232a452645cff2f5ccca221b6cc5b3b5a3789c9160" exitCode=0
Sep 11 14:42:20.932011 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:20.931673    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-ctd5d" event={"ID":"b35287e9-9950-4b9d-af5c-969dd954fead","Type":"ContainerDied","Data":"8e776083509e1e05204c4f232a452645cff2f5ccca221b6cc5b3b5a3789c9160"}
Sep 11 14:42:20.932053 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:20.932029    2559 scope.go:117] "RemoveContainer" containerID="8e776083509e1e05204c4f232a452645cff2f5ccca221b6cc5b3b5a3789c9160"
Sep 11 14:42:21.937595 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:21.937557    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-ctd5d" event={"ID":"b35287e9-9950-4b9d-af5c-969dd954fead","Type":"ContainerStarted","Data":"13dc7288f0c3f59939001b07390de9a2e2d10b718156903d2e5e7f00f4da14f8"}
Sep 11 14:42:24.133563 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:24.133527    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:42:24.196237 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:24.196201    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:42:33.948382 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:33.948317    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-659999cd6f-srzl2" podUID="b8943c90-72df-4971-8da6-d288d1f58f14" containerName="console" containerID="cri-o://768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e" gracePeriod=15
Sep 11 14:42:34.263688 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.263653    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:42:34.299980 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.299950    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-service-ca\") pod \"b8943c90-72df-4971-8da6-d288d1f58f14\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") "
Sep 11 14:42:34.300142 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.299991    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-oauth-config\") pod \"b8943c90-72df-4971-8da6-d288d1f58f14\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") "
Sep 11 14:42:34.300142 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.300025    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-trusted-ca-bundle\") pod \"b8943c90-72df-4971-8da6-d288d1f58f14\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") "
Sep 11 14:42:34.300142 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.300062    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbdzz\" (UniqueName: \"kubernetes.io/projected/b8943c90-72df-4971-8da6-d288d1f58f14-kube-api-access-cbdzz\") pod \"b8943c90-72df-4971-8da6-d288d1f58f14\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") "
Sep 11 14:42:34.300142 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.300079    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-console-config\") pod \"b8943c90-72df-4971-8da6-d288d1f58f14\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") "
Sep 11 14:42:34.300142 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.300099    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-oauth-serving-cert\") pod \"b8943c90-72df-4971-8da6-d288d1f58f14\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") "
Sep 11 14:42:34.300142 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.300117    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-serving-cert\") pod \"b8943c90-72df-4971-8da6-d288d1f58f14\" (UID: \"b8943c90-72df-4971-8da6-d288d1f58f14\") "
Sep 11 14:42:34.300526 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.300488    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b8943c90-72df-4971-8da6-d288d1f58f14" (UID: "b8943c90-72df-4971-8da6-d288d1f58f14"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:42:34.300793 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.300765    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b8943c90-72df-4971-8da6-d288d1f58f14" (UID: "b8943c90-72df-4971-8da6-d288d1f58f14"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:42:34.300883 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.300772    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-console-config" (OuterVolumeSpecName: "console-config") pod "b8943c90-72df-4971-8da6-d288d1f58f14" (UID: "b8943c90-72df-4971-8da6-d288d1f58f14"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:42:34.301000 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.300970    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-service-ca" (OuterVolumeSpecName: "service-ca") pod "b8943c90-72df-4971-8da6-d288d1f58f14" (UID: "b8943c90-72df-4971-8da6-d288d1f58f14"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:42:34.302679 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.302631    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b8943c90-72df-4971-8da6-d288d1f58f14" (UID: "b8943c90-72df-4971-8da6-d288d1f58f14"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:42:34.302679 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.302642    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b8943c90-72df-4971-8da6-d288d1f58f14" (UID: "b8943c90-72df-4971-8da6-d288d1f58f14"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:42:34.302809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.302683    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8943c90-72df-4971-8da6-d288d1f58f14-kube-api-access-cbdzz" (OuterVolumeSpecName: "kube-api-access-cbdzz") pod "b8943c90-72df-4971-8da6-d288d1f58f14" (UID: "b8943c90-72df-4971-8da6-d288d1f58f14"). InnerVolumeSpecName "kube-api-access-cbdzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:42:34.401393 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.401362    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:34.401393 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.401391    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-oauth-config\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:34.401605 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.401401    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-trusted-ca-bundle\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:34.401605 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.401412    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbdzz\" (UniqueName: \"kubernetes.io/projected/b8943c90-72df-4971-8da6-d288d1f58f14-kube-api-access-cbdzz\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:34.401605 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.401420    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-console-config\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:34.401605 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.401428    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8943c90-72df-4971-8da6-d288d1f58f14-oauth-serving-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:34.401605 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.401436    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8943c90-72df-4971-8da6-d288d1f58f14-console-serving-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:42:34.994107 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.994074    2559 generic.go:356] "Generic (PLEG): container finished" podID="b8943c90-72df-4971-8da6-d288d1f58f14" containerID="768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e" exitCode=0
Sep 11 14:42:34.994613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.994140    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-659999cd6f-srzl2"
Sep 11 14:42:34.994613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.994152    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-659999cd6f-srzl2" event={"ID":"b8943c90-72df-4971-8da6-d288d1f58f14","Type":"ContainerDied","Data":"768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e"}
Sep 11 14:42:34.994613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.994242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-659999cd6f-srzl2" event={"ID":"b8943c90-72df-4971-8da6-d288d1f58f14","Type":"ContainerDied","Data":"70d2ca3d05a120738d24eba7174351361f4faa334e4a8cf9b01c20a7447988f8"}
Sep 11 14:42:34.994613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:34.994269    2559 scope.go:117] "RemoveContainer" containerID="768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e"
Sep 11 14:42:35.004976 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:35.004950    2559 scope.go:117] "RemoveContainer" containerID="768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e"
Sep 11 14:42:35.005271 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:42:35.005250    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e\": container with ID starting with 768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e not found: ID does not exist" containerID="768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e"
Sep 11 14:42:35.005332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:35.005280    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e"} err="failed to get container status \"768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e\": rpc error: code = NotFound desc = could not find container \"768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e\": container with ID starting with 768fe699f5f1e631c2997325c22d26c74bf3ddb2324d57a29905aa80193ba58e not found: ID does not exist"
Sep 11 14:42:35.027964 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:35.027893    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-659999cd6f-srzl2"]
Sep 11 14:42:35.033804 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:35.033513    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-659999cd6f-srzl2"]
Sep 11 14:42:36.181805 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:36.181776    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8943c90-72df-4971-8da6-d288d1f58f14" path="/var/lib/kubelet/pods/b8943c90-72df-4971-8da6-d288d1f58f14/volumes"
Sep 11 14:42:55.961112 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:55.961064    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:42:55.980578 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:55.980549    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:42:56.097300 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:42:56.097273    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:43:05.566810 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:05.566774    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:05.636231 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:05.636192    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:14.119741 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:14.119695    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:14.211541 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:14.211475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:17.392524 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:17.392488    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:17.453246 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:17.453213    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:24.133207 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:24.133167    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:24.230572 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:24.230532    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:25.178563 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:25.178519    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65575bf675-4grsz"]
Sep 11 14:43:25.290565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:25.290534    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:25.391573 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:25.391510    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:25.468603 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:25.468503    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:50.205944 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.205881    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65575bf675-4grsz" podUID="d9057648-04d8-4caa-8dcc-085f2df0431c" containerName="console" containerID="cri-o://9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790" gracePeriod=15
Sep 11 14:43:50.450294 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.450264    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:43:50.525386 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.525345    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-964rg\" (UniqueName: \"kubernetes.io/projected/d9057648-04d8-4caa-8dcc-085f2df0431c-kube-api-access-964rg\") pod \"d9057648-04d8-4caa-8dcc-085f2df0431c\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") "
Sep 11 14:43:50.525608 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.525410    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-oauth-config\") pod \"d9057648-04d8-4caa-8dcc-085f2df0431c\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") "
Sep 11 14:43:50.525608 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.525482    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-service-ca\") pod \"d9057648-04d8-4caa-8dcc-085f2df0431c\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") "
Sep 11 14:43:50.525608 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.525527    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-trusted-ca-bundle\") pod \"d9057648-04d8-4caa-8dcc-085f2df0431c\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") "
Sep 11 14:43:50.525608 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.525550    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-console-config\") pod \"d9057648-04d8-4caa-8dcc-085f2df0431c\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") "
Sep 11 14:43:50.525608 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.525571    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-serving-cert\") pod \"d9057648-04d8-4caa-8dcc-085f2df0431c\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") "
Sep 11 14:43:50.525608 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.525598    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-oauth-serving-cert\") pod \"d9057648-04d8-4caa-8dcc-085f2df0431c\" (UID: \"d9057648-04d8-4caa-8dcc-085f2df0431c\") "
Sep 11 14:43:50.525893 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.525867    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-service-ca" (OuterVolumeSpecName: "service-ca") pod "d9057648-04d8-4caa-8dcc-085f2df0431c" (UID: "d9057648-04d8-4caa-8dcc-085f2df0431c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:43:50.525945 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.525901    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-console-config" (OuterVolumeSpecName: "console-config") pod "d9057648-04d8-4caa-8dcc-085f2df0431c" (UID: "d9057648-04d8-4caa-8dcc-085f2df0431c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:43:50.526022 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.525991    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d9057648-04d8-4caa-8dcc-085f2df0431c" (UID: "d9057648-04d8-4caa-8dcc-085f2df0431c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:43:50.526326 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.526298    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d9057648-04d8-4caa-8dcc-085f2df0431c" (UID: "d9057648-04d8-4caa-8dcc-085f2df0431c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:43:50.527616 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.527589    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d9057648-04d8-4caa-8dcc-085f2df0431c" (UID: "d9057648-04d8-4caa-8dcc-085f2df0431c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:43:50.527691 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.527623    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9057648-04d8-4caa-8dcc-085f2df0431c-kube-api-access-964rg" (OuterVolumeSpecName: "kube-api-access-964rg") pod "d9057648-04d8-4caa-8dcc-085f2df0431c" (UID: "d9057648-04d8-4caa-8dcc-085f2df0431c"). InnerVolumeSpecName "kube-api-access-964rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:43:50.527691 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.527671    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d9057648-04d8-4caa-8dcc-085f2df0431c" (UID: "d9057648-04d8-4caa-8dcc-085f2df0431c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:43:50.627037 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.627002    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-trusted-ca-bundle\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:43:50.627037 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.627033    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-console-config\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:43:50.627037 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.627045    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-serving-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:43:50.627289 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.627053    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-oauth-serving-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:43:50.627289 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.627064    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-964rg\" (UniqueName: \"kubernetes.io/projected/d9057648-04d8-4caa-8dcc-085f2df0431c-kube-api-access-964rg\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:43:50.627289 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.627072    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9057648-04d8-4caa-8dcc-085f2df0431c-console-oauth-config\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:43:50.627289 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:50.627080    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9057648-04d8-4caa-8dcc-085f2df0431c-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:43:51.288356 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:51.288316    2559 generic.go:356] "Generic (PLEG): container finished" podID="d9057648-04d8-4caa-8dcc-085f2df0431c" containerID="9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790" exitCode=0
Sep 11 14:43:51.288808 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:51.288367    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65575bf675-4grsz"
Sep 11 14:43:51.288808 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:51.288402    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65575bf675-4grsz" event={"ID":"d9057648-04d8-4caa-8dcc-085f2df0431c","Type":"ContainerDied","Data":"9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790"}
Sep 11 14:43:51.288808 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:51.288484    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65575bf675-4grsz" event={"ID":"d9057648-04d8-4caa-8dcc-085f2df0431c","Type":"ContainerDied","Data":"1f3e9280124356f1c6257409c4f3e6093fe23fd82f85240e327e81ada9d09d9b"}
Sep 11 14:43:51.288808 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:51.288506    2559 scope.go:117] "RemoveContainer" containerID="9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790"
Sep 11 14:43:51.296917 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:51.296901    2559 scope.go:117] "RemoveContainer" containerID="9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790"
Sep 11 14:43:51.297188 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:43:51.297170    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790\": container with ID starting with 9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790 not found: ID does not exist" containerID="9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790"
Sep 11 14:43:51.297247 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:51.297195    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790"} err="failed to get container status \"9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790\": rpc error: code = NotFound desc = could not find container \"9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790\": container with ID starting with 9377408457d0736020c84f192ccea4dcb4b72a5d2c09c92f83c8cdc01083a790 not found: ID does not exist"
Sep 11 14:43:51.311642 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:51.311612    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65575bf675-4grsz"]
Sep 11 14:43:51.315887 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:51.315857    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65575bf675-4grsz"]
Sep 11 14:43:51.828835 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:51.828798    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:51.884034 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:51.883984    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:43:52.178667 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:43:52.178594    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9057648-04d8-4caa-8dcc-085f2df0431c" path="/var/lib/kubelet/pods/d9057648-04d8-4caa-8dcc-085f2df0431c/volumes"
Sep 11 14:44:01.086739 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:01.086704    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:15.528908 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:15.528871    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:15.626617 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:15.626575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:21.513403 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:21.513321    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:21.598258 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:21.598225    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:37.804467 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:37.804405    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:38.543544 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:38.543504    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:41.300195 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:41.300156    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:41.401892 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:41.401858    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:44.796099 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:44.796065    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:44.958755 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:44.958721    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:45.218744 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:45.218649    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:45.478889 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:45.478848    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:47.080649 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:47.080615    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:47.337141 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:47.337037    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:47.574793 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:47.574758    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:47.935692 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:47.935659    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:53.883124 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:53.883086    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:54.107766 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:54.107741    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 14:44:56.181471 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:56.181419    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:56.275092 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:56.275045    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:56.906928 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:56.906893    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:44:57.003430 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:44:57.003399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:06.416581 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:06.416509    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:06.518257 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:06.518221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:08.349650 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:08.349614    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:22.692559 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:22.692521    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:23.944983 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.944950    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"]
Sep 11 14:45:23.945359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945314    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9057648-04d8-4caa-8dcc-085f2df0431c" containerName="console"
Sep 11 14:45:23.945359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945328    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9057648-04d8-4caa-8dcc-085f2df0431c" containerName="console"
Sep 11 14:45:23.945359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945340    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d2deea2-5322-4201-b2d4-08c50d4fb76e" containerName="console"
Sep 11 14:45:23.945359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945345    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d2deea2-5322-4201-b2d4-08c50d4fb76e" containerName="console"
Sep 11 14:45:23.945359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945352    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8943c90-72df-4971-8da6-d288d1f58f14" containerName="console"
Sep 11 14:45:23.945359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945358    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8943c90-72df-4971-8da6-d288d1f58f14" containerName="console"
Sep 11 14:45:23.945566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945374    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5e9fa73-5376-4267-a1d3-90625b5f0fd6" containerName="registry"
Sep 11 14:45:23.945566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945380    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5e9fa73-5376-4267-a1d3-90625b5f0fd6" containerName="registry"
Sep 11 14:45:23.945566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945430    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8943c90-72df-4971-8da6-d288d1f58f14" containerName="console"
Sep 11 14:45:23.945566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945438    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9057648-04d8-4caa-8dcc-085f2df0431c" containerName="console"
Sep 11 14:45:23.945566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945445    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5e9fa73-5376-4267-a1d3-90625b5f0fd6" containerName="registry"
Sep 11 14:45:23.945566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.945468    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d2deea2-5322-4201-b2d4-08c50d4fb76e" containerName="console"
Sep 11 14:45:23.948636 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.948612    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:23.953821 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.953764    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 14:45:23.954110 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.954091    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 14:45:23.954208 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.954160    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 14:45:23.954725 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.954707    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zjcsv\""
Sep 11 14:45:23.955647 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.955539    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 14:45:23.967375 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:23.967345    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"]
Sep 11 14:45:24.101719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.101677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-manifests\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.101924 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.101739    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kg7c\" (UniqueName: \"kubernetes.io/projected/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-kube-api-access-8kg7c\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.101924 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.101774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.101924 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.101820    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-tmp\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.101924 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.101860    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.202431 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.202320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-manifests\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.202431 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.202384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kg7c\" (UniqueName: \"kubernetes.io/projected/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-kube-api-access-8kg7c\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.202431 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.202418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.202762 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.202489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-tmp\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.202762 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.202515    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.202868 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.202811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-manifests\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.204881 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.204850    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-tmp\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.205048 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.205024    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.205092 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.205063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.211882 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.211853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kg7c\" (UniqueName: \"kubernetes.io/projected/165c4ad1-6229-42fc-a0dc-36db6cac1ffb-kube-api-access-8kg7c\") pod \"opendatahub-operator-controller-manager-67f987f856-28tv4\" (UID: \"165c4ad1-6229-42fc-a0dc-36db6cac1ffb\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.257654 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.257616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:24.259433 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.259412    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:24.358541 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.358509    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:24.459642 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.459550    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:24.552937 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.552909    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"]
Sep 11 14:45:24.555093 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:45:24.555063    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165c4ad1_6229_42fc_a0dc_36db6cac1ffb.slice/crio-331ffd2ce11f2a0555e412e0d2b8549ee8faff33ee244714c195c9a9a22558ea WatchSource:0}: Error finding container 331ffd2ce11f2a0555e412e0d2b8549ee8faff33ee244714c195c9a9a22558ea: Status 404 returned error can't find the container with id 331ffd2ce11f2a0555e412e0d2b8549ee8faff33ee244714c195c9a9a22558ea
Sep 11 14:45:24.556951 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.556928    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:45:24.569342 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.569314    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:24.640303 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.640268    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4" event={"ID":"165c4ad1-6229-42fc-a0dc-36db6cac1ffb","Type":"ContainerStarted","Data":"331ffd2ce11f2a0555e412e0d2b8549ee8faff33ee244714c195c9a9a22558ea"}
Sep 11 14:45:24.658369 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.658334    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:24.751752 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:24.751715    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:27.656500 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:27.656435    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4" event={"ID":"165c4ad1-6229-42fc-a0dc-36db6cac1ffb","Type":"ContainerStarted","Data":"c0d812c9f6d22f8a47a69531df7ce94f3e6fc2842f867ed82e287eba7deeced0"}
Sep 11 14:45:29.666894 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:29.666855    2559 generic.go:356] "Generic (PLEG): container finished" podID="165c4ad1-6229-42fc-a0dc-36db6cac1ffb" containerID="c0d812c9f6d22f8a47a69531df7ce94f3e6fc2842f867ed82e287eba7deeced0" exitCode=0
Sep 11 14:45:29.667330 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:29.666911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4" event={"ID":"165c4ad1-6229-42fc-a0dc-36db6cac1ffb","Type":"ContainerDied","Data":"c0d812c9f6d22f8a47a69531df7ce94f3e6fc2842f867ed82e287eba7deeced0"}
Sep 11 14:45:30.672734 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:30.672698    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4" event={"ID":"165c4ad1-6229-42fc-a0dc-36db6cac1ffb","Type":"ContainerStarted","Data":"6cf2ad4adda88c909a69a852111bc37a7ff37de57208108b3f55990faf03fd3b"}
Sep 11 14:45:30.673133 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:30.672925    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:30.696727 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:30.696669    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4" podStartSLOduration=3.255307054 podStartE2EDuration="7.696645029s" totalImagesPullingTime="3.351263662s" totalInitContainerRuntime="1.090074313s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:30.695347176 +0000 UTC m=+337.017975570" watchObservedRunningTime="2026-09-11 14:45:30.696645029 +0000 UTC m=+337.019273398"
Sep 11 14:45:40.590882 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:40.590842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:40.696989 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:40.696947    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:41.679492 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:41.679443    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-28tv4"
Sep 11 14:45:42.608500 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.608464    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"]
Sep 11 14:45:42.621177 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.621146    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.625786 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.625759    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 14:45:42.625786 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.625763    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 14:45:42.625999 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.625832    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:45:42.625999 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.625763    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j64lt\""
Sep 11 14:45:42.625999 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.625879    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 14:45:42.630544 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.630520    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 14:45:42.640359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.640322    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"]
Sep 11 14:45:42.651881 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.651841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-cert\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.652053 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.651897    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-manager-config\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.652053 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.651966    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-metrics-cert\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.652165 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.652072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snwch\" (UniqueName: \"kubernetes.io/projected/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-kube-api-access-snwch\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.752711 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.752672    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-cert\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.752711 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.752718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-manager-config\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.753260 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.752738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-metrics-cert\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.753260 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.752781    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snwch\" (UniqueName: \"kubernetes.io/projected/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-kube-api-access-snwch\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.753466 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.753429    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-manager-config\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.755145 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.755121    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-metrics-cert\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.755262 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.755239    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-cert\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.769364 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.769338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snwch\" (UniqueName: \"kubernetes.io/projected/23913e6b-0ce6-4e0a-bd7a-924ea30c5b65-kube-api-access-snwch\") pod \"lws-controller-manager-6c79f8858d-crmtv\" (UID: \"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:42.930151 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:42.930049    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:43.211768 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.211732    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"]
Sep 11 14:45:43.214444 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:45:43.214408    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23913e6b_0ce6_4e0a_bd7a_924ea30c5b65.slice/crio-d02f71ce97031f60941656697bd69c73423212dfaa6c54ae1bc34de1c3499371 WatchSource:0}: Error finding container d02f71ce97031f60941656697bd69c73423212dfaa6c54ae1bc34de1c3499371: Status 404 returned error can't find the container with id d02f71ce97031f60941656697bd69c73423212dfaa6c54ae1bc34de1c3499371
Sep 11 14:45:43.288604 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.288551    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:43.464774 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.464683    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:43.728722 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.728688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv" event={"ID":"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65","Type":"ContainerStarted","Data":"d02f71ce97031f60941656697bd69c73423212dfaa6c54ae1bc34de1c3499371"}
Sep 11 14:45:43.772006 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.771964    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:43.909573 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.909539    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-789999dc4-tcljc"]
Sep 11 14:45:43.913985 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.913958    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc"
Sep 11 14:45:43.919945 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.919917    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 14:45:43.921253 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.921231    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bp2n9\""
Sep 11 14:45:43.921366 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.921280    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 14:45:43.921428 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.921387    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 14:45:43.921497 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.921386    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 14:45:43.932484 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.932417    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-789999dc4-tcljc"]
Sep 11 14:45:43.959521 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.959477    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:43.961647 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.961606    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b3ac8f-8c9a-4b52-8252-ce369f6f1e18-tls-certs\") pod \"kube-auth-proxy-789999dc4-tcljc\" (UID: \"36b3ac8f-8c9a-4b52-8252-ce369f6f1e18\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc"
Sep 11 14:45:43.961812 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.961712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36b3ac8f-8c9a-4b52-8252-ce369f6f1e18-tmp\") pod \"kube-auth-proxy-789999dc4-tcljc\" (UID: \"36b3ac8f-8c9a-4b52-8252-ce369f6f1e18\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc"
Sep 11 14:45:43.961812 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:43.961735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk64c\" (UniqueName: \"kubernetes.io/projected/36b3ac8f-8c9a-4b52-8252-ce369f6f1e18-kube-api-access-pk64c\") pod \"kube-auth-proxy-789999dc4-tcljc\" (UID: \"36b3ac8f-8c9a-4b52-8252-ce369f6f1e18\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc"
Sep 11 14:45:44.062792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:44.062651    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b3ac8f-8c9a-4b52-8252-ce369f6f1e18-tls-certs\") pod \"kube-auth-proxy-789999dc4-tcljc\" (UID: \"36b3ac8f-8c9a-4b52-8252-ce369f6f1e18\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc"
Sep 11 14:45:44.063090 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:44.063061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36b3ac8f-8c9a-4b52-8252-ce369f6f1e18-tmp\") pod \"kube-auth-proxy-789999dc4-tcljc\" (UID: \"36b3ac8f-8c9a-4b52-8252-ce369f6f1e18\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc"
Sep 11 14:45:44.063233 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:44.063102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk64c\" (UniqueName: \"kubernetes.io/projected/36b3ac8f-8c9a-4b52-8252-ce369f6f1e18-kube-api-access-pk64c\") pod \"kube-auth-proxy-789999dc4-tcljc\" (UID: \"36b3ac8f-8c9a-4b52-8252-ce369f6f1e18\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc"
Sep 11 14:45:44.065958 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:44.065927    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36b3ac8f-8c9a-4b52-8252-ce369f6f1e18-tmp\") pod \"kube-auth-proxy-789999dc4-tcljc\" (UID: \"36b3ac8f-8c9a-4b52-8252-ce369f6f1e18\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc"
Sep 11 14:45:44.066228 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:44.066186    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b3ac8f-8c9a-4b52-8252-ce369f6f1e18-tls-certs\") pod \"kube-auth-proxy-789999dc4-tcljc\" (UID: \"36b3ac8f-8c9a-4b52-8252-ce369f6f1e18\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc"
Sep 11 14:45:44.075364 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:44.075307    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk64c\" (UniqueName: \"kubernetes.io/projected/36b3ac8f-8c9a-4b52-8252-ce369f6f1e18-kube-api-access-pk64c\") pod \"kube-auth-proxy-789999dc4-tcljc\" (UID: \"36b3ac8f-8c9a-4b52-8252-ce369f6f1e18\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc"
Sep 11 14:45:44.223688 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:44.223655    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc"
Sep 11 14:45:44.533950 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:44.533913    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-789999dc4-tcljc"]
Sep 11 14:45:44.537196 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:45:44.537166    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b3ac8f_8c9a_4b52_8252_ce369f6f1e18.slice/crio-f02521cc7ba31c571b6d7fe8b3fe62e73c280c20813f5ac2323f8d0d89a8f7b1 WatchSource:0}: Error finding container f02521cc7ba31c571b6d7fe8b3fe62e73c280c20813f5ac2323f8d0d89a8f7b1: Status 404 returned error can't find the container with id f02521cc7ba31c571b6d7fe8b3fe62e73c280c20813f5ac2323f8d0d89a8f7b1
Sep 11 14:45:44.738322 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:44.738286    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc" event={"ID":"36b3ac8f-8c9a-4b52-8252-ce369f6f1e18","Type":"ContainerStarted","Data":"f02521cc7ba31c571b6d7fe8b3fe62e73c280c20813f5ac2323f8d0d89a8f7b1"}
Sep 11 14:45:45.454184 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:45.454116    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:45.655869 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:45.655835    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:45.886400 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:45.886360    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:45.977149 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:45.977111    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:46.752003 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:46.751953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv" event={"ID":"23913e6b-0ce6-4e0a-bd7a-924ea30c5b65","Type":"ContainerStarted","Data":"f1c513736909867b8c661aa11dcefdccc21121fe88582c4eb0db280a48b505a7"}
Sep 11 14:45:46.752528 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:46.752111    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:45:46.771442 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:46.771367    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv" podStartSLOduration=2.319855356 podStartE2EDuration="4.771345792s" totalImagesPullingTime="2.451490436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:46.770589828 +0000 UTC m=+353.093218197" watchObservedRunningTime="2026-09-11 14:45:46.771345792 +0000 UTC m=+353.093974161"
Sep 11 14:45:47.289415 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:47.289347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:48.762571 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:48.762531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc" event={"ID":"36b3ac8f-8c9a-4b52-8252-ce369f6f1e18","Type":"ContainerStarted","Data":"b8ec34c172c3b1901fc73b205016ccf3f7577675c4979fc4d0ac862ba2ba005b"}
Sep 11 14:45:48.780769 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:48.780715    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-789999dc4-tcljc" podStartSLOduration=2.43519344 podStartE2EDuration="5.780696065s" totalImagesPullingTime="3.345502625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:48.779481236 +0000 UTC m=+355.102109607" watchObservedRunningTime="2026-09-11 14:45:48.780696065 +0000 UTC m=+355.103324434"
Sep 11 14:45:51.280064 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:51.280029    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:51.381238 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:51.381196    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:51.960390 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:51.960354    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:52.193823 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:52.193783    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:52.417957 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:52.417924    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:54.714195 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:54.714157    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:54.784401 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:54.784359    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:54.884143 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:54.884106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:55.264042 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:55.264001    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:55.425732 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:55.425698    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:55.641443 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:55.641348    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:45:57.759324 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:45:57.759267    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-crmtv"
Sep 11 14:46:05.288073 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:05.287981    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:05.395901 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:05.395864    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:05.523657 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:05.523612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:05.638770 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:05.638662    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:06.585756 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:06.585720    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:06.686323 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:06.686290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:06.839412 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:06.839309    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:38.602059 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.602016    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-596d89868f-n4j2s"]
Sep 11 14:46:38.610983 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.610949    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.613936 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.613903    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 14:46:38.614103 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.613903    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 14:46:38.614103 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.613975    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 14:46:38.615278 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.615227    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 14:46:38.615278 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.615241    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 14:46:38.615486 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.615291    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8v4w7\""
Sep 11 14:46:38.615486 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.615352    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 14:46:38.616470 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.616411    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 14:46:38.620086 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.620061    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596d89868f-n4j2s"]
Sep 11 14:46:38.620521 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.620501    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 14:46:38.638579 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.638539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-trusted-ca-bundle\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.638579 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.638580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-oauth-serving-cert\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.638836 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.638660    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beac3c4c-6e77-4706-99cc-818b3d6d44e3-console-oauth-config\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.638836 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.638727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beac3c4c-6e77-4706-99cc-818b3d6d44e3-console-serving-cert\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.638836 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.638755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-console-config\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.638836 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.638785    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-service-ca\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.638993 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.638845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgghj\" (UniqueName: \"kubernetes.io/projected/beac3c4c-6e77-4706-99cc-818b3d6d44e3-kube-api-access-vgghj\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.693819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.693779    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:38.740045 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.740009    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgghj\" (UniqueName: \"kubernetes.io/projected/beac3c4c-6e77-4706-99cc-818b3d6d44e3-kube-api-access-vgghj\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.740225 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.740052    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-trusted-ca-bundle\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.740225 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.740081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-oauth-serving-cert\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.740225 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.740136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beac3c4c-6e77-4706-99cc-818b3d6d44e3-console-oauth-config\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.740225 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.740195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beac3c4c-6e77-4706-99cc-818b3d6d44e3-console-serving-cert\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.740225 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.740217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-console-config\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.740520 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.740243    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-service-ca\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.741035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.741007    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-oauth-serving-cert\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.741035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.741021    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-trusted-ca-bundle\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.741205 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.741038    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-service-ca\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.741205 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.741011    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beac3c4c-6e77-4706-99cc-818b3d6d44e3-console-config\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.742771 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.742745    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beac3c4c-6e77-4706-99cc-818b3d6d44e3-console-serving-cert\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.742854 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.742775    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beac3c4c-6e77-4706-99cc-818b3d6d44e3-console-oauth-config\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.748921 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.748896    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgghj\" (UniqueName: \"kubernetes.io/projected/beac3c4c-6e77-4706-99cc-818b3d6d44e3-kube-api-access-vgghj\") pod \"console-596d89868f-n4j2s\" (UID: \"beac3c4c-6e77-4706-99cc-818b3d6d44e3\") " pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:38.922723 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:38.922619    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:39.200906 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:39.200826    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596d89868f-n4j2s"]
Sep 11 14:46:39.203995 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:46:39.203964    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeac3c4c_6e77_4706_99cc_818b3d6d44e3.slice/crio-2d53f55533bdd88fba394b27dbc859eca4bc7331ca7f40ca6c2a35e244b91131 WatchSource:0}: Error finding container 2d53f55533bdd88fba394b27dbc859eca4bc7331ca7f40ca6c2a35e244b91131: Status 404 returned error can't find the container with id 2d53f55533bdd88fba394b27dbc859eca4bc7331ca7f40ca6c2a35e244b91131
Sep 11 14:46:39.991024 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:39.990989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596d89868f-n4j2s" event={"ID":"beac3c4c-6e77-4706-99cc-818b3d6d44e3","Type":"ContainerStarted","Data":"3b4f60d1e42af9ef21a84c44bf0cc29518622c7b99f701bfe7a6eabf111cfcef"}
Sep 11 14:46:39.991466 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:39.991040    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596d89868f-n4j2s" event={"ID":"beac3c4c-6e77-4706-99cc-818b3d6d44e3","Type":"ContainerStarted","Data":"2d53f55533bdd88fba394b27dbc859eca4bc7331ca7f40ca6c2a35e244b91131"}
Sep 11 14:46:40.014629 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:40.014567    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-596d89868f-n4j2s" podStartSLOduration=2.014547126 podStartE2EDuration="2.014547126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:46:40.013049418 +0000 UTC m=+406.335677786" watchObservedRunningTime="2026-09-11 14:46:40.014547126 +0000 UTC m=+406.337175494"
Sep 11 14:46:48.923610 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:48.923551    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:48.924028 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:48.923631    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:48.928511 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:48.928471    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:49.034362 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:49.034334    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-596d89868f-n4j2s"
Sep 11 14:46:49.206259 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:49.206166    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:49.261108 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:49.261077    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:49.319817 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:49.319760    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:55.588985 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:55.588948    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:46:55.708311 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:46:55.708271    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:08.368566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:08.368527    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:09.162949 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:09.162909    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:12.362710 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:12.362676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:12.727212 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:12.727146    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:14.211543 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:14.211511    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:14.650353 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:14.650317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:15.372806 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:15.372762    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:15.960958 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:15.960930    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:16.393001 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:16.392967    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:16.520037 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:16.519950    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:20.030590 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.030555    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"]
Sep 11 14:47:20.033904 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.033882    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:20.037132 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.037110    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 14:47:20.037287 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.037111    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 14:47:20.037618 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.037604    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qns29\""
Sep 11 14:47:20.061867 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.061827    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"]
Sep 11 14:47:20.208788 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.208752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zpz6t\" (UID: \"5be945ac-bbcb-4ce6-924b-9dbaa32781fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:20.208969 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.208840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62twm\" (UniqueName: \"kubernetes.io/projected/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-kube-api-access-62twm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zpz6t\" (UID: \"5be945ac-bbcb-4ce6-924b-9dbaa32781fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:20.310009 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.309909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62twm\" (UniqueName: \"kubernetes.io/projected/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-kube-api-access-62twm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zpz6t\" (UID: \"5be945ac-bbcb-4ce6-924b-9dbaa32781fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:20.310009 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.309974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zpz6t\" (UID: \"5be945ac-bbcb-4ce6-924b-9dbaa32781fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:20.310362 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.310340    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zpz6t\" (UID: \"5be945ac-bbcb-4ce6-924b-9dbaa32781fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:20.321564 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.321539    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62twm\" (UniqueName: \"kubernetes.io/projected/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-kube-api-access-62twm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zpz6t\" (UID: \"5be945ac-bbcb-4ce6-924b-9dbaa32781fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:20.344437 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.344377    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:20.626407 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.626323    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"]
Sep 11 14:47:20.629907 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:47:20.629879    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be945ac_bbcb_4ce6_924b_9dbaa32781fd.slice/crio-033969be9faed5f6a9de8244742fa069c41b689e00d68965c2aa892782c923d1 WatchSource:0}: Error finding container 033969be9faed5f6a9de8244742fa069c41b689e00d68965c2aa892782c923d1: Status 404 returned error can't find the container with id 033969be9faed5f6a9de8244742fa069c41b689e00d68965c2aa892782c923d1
Sep 11 14:47:20.969939 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:20.969843    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:21.176279 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:21.176244    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t" event={"ID":"5be945ac-bbcb-4ce6-924b-9dbaa32781fd","Type":"ContainerStarted","Data":"033969be9faed5f6a9de8244742fa069c41b689e00d68965c2aa892782c923d1"}
Sep 11 14:47:23.269952 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:23.269911    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:24.775026 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:24.774979    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:24.865253 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:24.865216    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:25.186652 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:25.186554    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:25.200176 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:25.200140    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t" event={"ID":"5be945ac-bbcb-4ce6-924b-9dbaa32781fd","Type":"ContainerStarted","Data":"fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b"}
Sep 11 14:47:25.200364 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:25.200347    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:25.231640 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:25.231564    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t" podStartSLOduration=1.4421136589999999 podStartE2EDuration="5.231543127s" totalImagesPullingTime="3.789429468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:47:25.231071096 +0000 UTC m=+451.553699465" watchObservedRunningTime="2026-09-11 14:47:25.231543127 +0000 UTC m=+451.554171501"
Sep 11 14:47:25.296550 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:25.296507    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:27.207658 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:27.207628    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:27.325992 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:27.325955    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:28.571878 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:28.571832    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:28.675136 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:28.675094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:30.552794 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:30.552743    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:30.618508 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:30.618444    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:33.540906 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:33.540866    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:33.640856 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:33.640817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:33.739414 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:33.739379    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:36.206359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:36.206319    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:38.478337 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.478300    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"]
Sep 11 14:47:38.478824 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.478487    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t" podUID="5be945ac-bbcb-4ce6-924b-9dbaa32781fd" containerName="manager" containerID="cri-o://fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b" gracePeriod=2
Sep 11 14:47:38.486110 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.486074    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"]
Sep 11 14:47:38.509865 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.509831    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"]
Sep 11 14:47:38.510186 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.510172    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be945ac-bbcb-4ce6-924b-9dbaa32781fd" containerName="manager"
Sep 11 14:47:38.510186 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.510185    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be945ac-bbcb-4ce6-924b-9dbaa32781fd" containerName="manager"
Sep 11 14:47:38.510327 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.510263    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be945ac-bbcb-4ce6-924b-9dbaa32781fd" containerName="manager"
Sep 11 14:47:38.515204 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.515156    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:38.528513 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.528487    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"]
Sep 11 14:47:38.566067 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:47:38.566031    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zpz6t\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="5be945ac-bbcb-4ce6-924b-9dbaa32781fd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:38.573259 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.571792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxgd\" (UniqueName: \"kubernetes.io/projected/df6089b8-9ef4-43d1-a103-221c3e3a428e-kube-api-access-kzxgd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-c2kvm\" (UID: \"df6089b8-9ef4-43d1-a103-221c3e3a428e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:38.573259 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.571913    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df6089b8-9ef4-43d1-a103-221c3e3a428e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-c2kvm\" (UID: \"df6089b8-9ef4-43d1-a103-221c3e3a428e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:38.673073 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.673027    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxgd\" (UniqueName: \"kubernetes.io/projected/df6089b8-9ef4-43d1-a103-221c3e3a428e-kube-api-access-kzxgd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-c2kvm\" (UID: \"df6089b8-9ef4-43d1-a103-221c3e3a428e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:38.673245 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.673102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df6089b8-9ef4-43d1-a103-221c3e3a428e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-c2kvm\" (UID: \"df6089b8-9ef4-43d1-a103-221c3e3a428e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:38.673493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.673469    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df6089b8-9ef4-43d1-a103-221c3e3a428e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-c2kvm\" (UID: \"df6089b8-9ef4-43d1-a103-221c3e3a428e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:38.688370 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.688333    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxgd\" (UniqueName: \"kubernetes.io/projected/df6089b8-9ef4-43d1-a103-221c3e3a428e-kube-api-access-kzxgd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-c2kvm\" (UID: \"df6089b8-9ef4-43d1-a103-221c3e3a428e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:38.725252 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.725228    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:38.727874 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:47:38.727847    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zpz6t\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="5be945ac-bbcb-4ce6-924b-9dbaa32781fd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:38.774433 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.774347    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-extensions-socket-volume\") pod \"5be945ac-bbcb-4ce6-924b-9dbaa32781fd\" (UID: \"5be945ac-bbcb-4ce6-924b-9dbaa32781fd\") "
Sep 11 14:47:38.774433 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.774398    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62twm\" (UniqueName: \"kubernetes.io/projected/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-kube-api-access-62twm\") pod \"5be945ac-bbcb-4ce6-924b-9dbaa32781fd\" (UID: \"5be945ac-bbcb-4ce6-924b-9dbaa32781fd\") "
Sep 11 14:47:38.774923 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.774895    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5be945ac-bbcb-4ce6-924b-9dbaa32781fd" (UID: "5be945ac-bbcb-4ce6-924b-9dbaa32781fd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:38.776660 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.776640    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-kube-api-access-62twm" (OuterVolumeSpecName: "kube-api-access-62twm") pod "5be945ac-bbcb-4ce6-924b-9dbaa32781fd" (UID: "5be945ac-bbcb-4ce6-924b-9dbaa32781fd"). InnerVolumeSpecName "kube-api-access-62twm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:38.872742 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.872700    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:38.875053 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.875030    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-extensions-socket-volume\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:47:38.875107 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:38.875056    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62twm\" (UniqueName: \"kubernetes.io/projected/5be945ac-bbcb-4ce6-924b-9dbaa32781fd-kube-api-access-62twm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:47:39.156335 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:39.156310    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"]
Sep 11 14:47:39.158771 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:47:39.158735    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6089b8_9ef4_43d1_a103_221c3e3a428e.slice/crio-de5645121e1f93e4c723e70dde8315217d0226a820a45cfcff38da64bb0b6380 WatchSource:0}: Error finding container de5645121e1f93e4c723e70dde8315217d0226a820a45cfcff38da64bb0b6380: Status 404 returned error can't find the container with id de5645121e1f93e4c723e70dde8315217d0226a820a45cfcff38da64bb0b6380
Sep 11 14:47:39.262991 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:39.262959    2559 generic.go:356] "Generic (PLEG): container finished" podID="5be945ac-bbcb-4ce6-924b-9dbaa32781fd" containerID="fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b" exitCode=0
Sep 11 14:47:39.263206 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:39.263012    2559 scope.go:117] "RemoveContainer" containerID="fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b"
Sep 11 14:47:39.263206 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:39.263026    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:39.265243 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:39.265220    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm" event={"ID":"df6089b8-9ef4-43d1-a103-221c3e3a428e","Type":"ContainerStarted","Data":"bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc"}
Sep 11 14:47:39.265368 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:39.265269    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm" event={"ID":"df6089b8-9ef4-43d1-a103-221c3e3a428e","Type":"ContainerStarted","Data":"de5645121e1f93e4c723e70dde8315217d0226a820a45cfcff38da64bb0b6380"}
Sep 11 14:47:39.265423 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:39.265378    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:39.266527 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:47:39.266497    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zpz6t\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="5be945ac-bbcb-4ce6-924b-9dbaa32781fd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:39.275629 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:39.275610    2559 scope.go:117] "RemoveContainer" containerID="fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b"
Sep 11 14:47:39.275974 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:47:39.275945    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b\": container with ID starting with fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b not found: ID does not exist" containerID="fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b"
Sep 11 14:47:39.276023 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:39.275984    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b"} err="failed to get container status \"fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b\": rpc error: code = NotFound desc = could not find container \"fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b\": container with ID starting with fc47a7b7177df1363eed9fe2350fe8082f8420d78df442f8d3f000b604c81c1b not found: ID does not exist"
Sep 11 14:47:39.312569 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:39.312516    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm" podStartSLOduration=1.312500377 podStartE2EDuration="1.312500377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:47:39.310418993 +0000 UTC m=+465.633047361" watchObservedRunningTime="2026-09-11 14:47:39.312500377 +0000 UTC m=+465.635128745"
Sep 11 14:47:39.314369 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:47:39.314340    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zpz6t\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="5be945ac-bbcb-4ce6-924b-9dbaa32781fd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:39.316957 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:47:39.316932    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zpz6t\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="5be945ac-bbcb-4ce6-924b-9dbaa32781fd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zpz6t"
Sep 11 14:47:40.180789 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:40.180748    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5be945ac-bbcb-4ce6-924b-9dbaa32781fd" path="/var/lib/kubelet/pods/5be945ac-bbcb-4ce6-924b-9dbaa32781fd/volumes"
Sep 11 14:47:40.634226 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:40.634179    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:40.742641 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:40.742594    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:43.660661 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:43.660624    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:43.761542 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:43.761475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:47.454177 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:47.454134    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:47.556955 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:47.556906    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:50.272984 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:50.272952    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:53.432040 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:53.432005    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:53.504553 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:53.504513    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:53.799940 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:53.799910    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"]
Sep 11 14:47:53.800116 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:53.800086    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm" podUID="df6089b8-9ef4-43d1-a103-221c3e3a428e" containerName="manager" containerID="cri-o://bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc" gracePeriod=10
Sep 11 14:47:54.046102 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.046077    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:54.081319 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.081232    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:54.104803 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.104767    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"]
Sep 11 14:47:54.105111 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.105099    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6089b8-9ef4-43d1-a103-221c3e3a428e" containerName="manager"
Sep 11 14:47:54.105152 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.105112    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6089b8-9ef4-43d1-a103-221c3e3a428e" containerName="manager"
Sep 11 14:47:54.105186 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.105162    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6089b8-9ef4-43d1-a103-221c3e3a428e" containerName="manager"
Sep 11 14:47:54.108094 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.108074    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 14:47:54.129894 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.129850    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"]
Sep 11 14:47:54.189649 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.189612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:54.207618 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.207581    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzxgd\" (UniqueName: \"kubernetes.io/projected/df6089b8-9ef4-43d1-a103-221c3e3a428e-kube-api-access-kzxgd\") pod \"df6089b8-9ef4-43d1-a103-221c3e3a428e\" (UID: \"df6089b8-9ef4-43d1-a103-221c3e3a428e\") "
Sep 11 14:47:54.207792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.207638    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df6089b8-9ef4-43d1-a103-221c3e3a428e-extensions-socket-volume\") pod \"df6089b8-9ef4-43d1-a103-221c3e3a428e\" (UID: \"df6089b8-9ef4-43d1-a103-221c3e3a428e\") "
Sep 11 14:47:54.207792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.207774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bff8j\" (UniqueName: \"kubernetes.io/projected/e61d46f5-9367-4e10-90bb-3aadda2a5486-kube-api-access-bff8j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r6m6n\" (UID: \"e61d46f5-9367-4e10-90bb-3aadda2a5486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 14:47:54.207915 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.207812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e61d46f5-9367-4e10-90bb-3aadda2a5486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r6m6n\" (UID: \"e61d46f5-9367-4e10-90bb-3aadda2a5486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 14:47:54.208030 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.208006    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6089b8-9ef4-43d1-a103-221c3e3a428e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "df6089b8-9ef4-43d1-a103-221c3e3a428e" (UID: "df6089b8-9ef4-43d1-a103-221c3e3a428e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:54.209728 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.209706    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6089b8-9ef4-43d1-a103-221c3e3a428e-kube-api-access-kzxgd" (OuterVolumeSpecName: "kube-api-access-kzxgd") pod "df6089b8-9ef4-43d1-a103-221c3e3a428e" (UID: "df6089b8-9ef4-43d1-a103-221c3e3a428e"). InnerVolumeSpecName "kube-api-access-kzxgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:54.308589 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.308553    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bff8j\" (UniqueName: \"kubernetes.io/projected/e61d46f5-9367-4e10-90bb-3aadda2a5486-kube-api-access-bff8j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r6m6n\" (UID: \"e61d46f5-9367-4e10-90bb-3aadda2a5486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 14:47:54.308755 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.308614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e61d46f5-9367-4e10-90bb-3aadda2a5486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r6m6n\" (UID: \"e61d46f5-9367-4e10-90bb-3aadda2a5486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 14:47:54.308755 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.308672    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzxgd\" (UniqueName: \"kubernetes.io/projected/df6089b8-9ef4-43d1-a103-221c3e3a428e-kube-api-access-kzxgd\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:47:54.308755 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.308683    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df6089b8-9ef4-43d1-a103-221c3e3a428e-extensions-socket-volume\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:47:54.308999 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.308978    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e61d46f5-9367-4e10-90bb-3aadda2a5486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r6m6n\" (UID: \"e61d46f5-9367-4e10-90bb-3aadda2a5486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 14:47:54.317556 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.317530    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bff8j\" (UniqueName: \"kubernetes.io/projected/e61d46f5-9367-4e10-90bb-3aadda2a5486-kube-api-access-bff8j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r6m6n\" (UID: \"e61d46f5-9367-4e10-90bb-3aadda2a5486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 14:47:54.339584 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.339489    2559 generic.go:356] "Generic (PLEG): container finished" podID="df6089b8-9ef4-43d1-a103-221c3e3a428e" containerID="bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc" exitCode=0
Sep 11 14:47:54.339584 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.339540    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"
Sep 11 14:47:54.339755 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.339582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm" event={"ID":"df6089b8-9ef4-43d1-a103-221c3e3a428e","Type":"ContainerDied","Data":"bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc"}
Sep 11 14:47:54.339755 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.339632    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm" event={"ID":"df6089b8-9ef4-43d1-a103-221c3e3a428e","Type":"ContainerDied","Data":"de5645121e1f93e4c723e70dde8315217d0226a820a45cfcff38da64bb0b6380"}
Sep 11 14:47:54.339755 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.339655    2559 scope.go:117] "RemoveContainer" containerID="bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc"
Sep 11 14:47:54.348610 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.348590    2559 scope.go:117] "RemoveContainer" containerID="bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc"
Sep 11 14:47:54.348904 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:47:54.348882    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc\": container with ID starting with bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc not found: ID does not exist" containerID="bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc"
Sep 11 14:47:54.348971 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.348912    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc"} err="failed to get container status \"bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc\": rpc error: code = NotFound desc = could not find container \"bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc\": container with ID starting with bead7479b5887012f9d319d8a0c73a2e922e98d8ece00853f8af941916b7bcfc not found: ID does not exist"
Sep 11 14:47:54.370336 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.370296    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"]
Sep 11 14:47:54.374701 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.374673    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-c2kvm"]
Sep 11 14:47:54.383903 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.383864    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:54.417982 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.417950    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 14:47:54.621032 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.620942    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:54.721186 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.721157    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"]
Sep 11 14:47:54.731439 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:47:54.731412    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61d46f5_9367_4e10_90bb_3aadda2a5486.slice/crio-df924e051c740c1b663cb2acf0983f5e238a0424eb1d990fae548c8ec6cb6ff7 WatchSource:0}: Error finding container df924e051c740c1b663cb2acf0983f5e238a0424eb1d990fae548c8ec6cb6ff7: Status 404 returned error can't find the container with id df924e051c740c1b663cb2acf0983f5e238a0424eb1d990fae548c8ec6cb6ff7
Sep 11 14:47:54.753748 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:54.753699    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:47:55.347853 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:55.347805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n" event={"ID":"e61d46f5-9367-4e10-90bb-3aadda2a5486","Type":"ContainerStarted","Data":"ed92f160316592d1ad173168b9bd6130c6c345c6f77f18ce0f20ed7d4913f6bd"}
Sep 11 14:47:55.347853 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:55.347857    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n" event={"ID":"e61d46f5-9367-4e10-90bb-3aadda2a5486","Type":"ContainerStarted","Data":"df924e051c740c1b663cb2acf0983f5e238a0424eb1d990fae548c8ec6cb6ff7"}
Sep 11 14:47:55.348073 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:55.347947    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 14:47:55.379246 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:55.379168    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n" podStartSLOduration=1.379150102 podStartE2EDuration="1.379150102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:47:55.378664616 +0000 UTC m=+481.701292983" watchObservedRunningTime="2026-09-11 14:47:55.379150102 +0000 UTC m=+481.701778452"
Sep 11 14:47:56.179569 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:47:56.179537    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df6089b8-9ef4-43d1-a103-221c3e3a428e" path="/var/lib/kubelet/pods/df6089b8-9ef4-43d1-a103-221c3e3a428e/volumes"
Sep 11 14:48:01.627919 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:01.627881    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:01.727886 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:01.727848    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:06.354838 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:06.354809    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 14:48:10.300005 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:10.299965    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:10.549366 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:10.549323    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:10.906853 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:10.906821    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:21.890022 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:21.889981    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:22.021648 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:22.021617    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:26.166104 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.166066    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:26.269406 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.269364    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:26.341943 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.341904    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:26.775688 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.775654    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:48:26.779100 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.779080    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"
Sep 11 14:48:26.783390 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.783365    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6xbxv\""
Sep 11 14:48:26.783585 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.783566    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 14:48:26.790804 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.790779    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:48:26.850526 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.850492    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:48:26.891370 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.891336    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6aaec257-03ee-4ee4-9490-8305bf1ac755-config-file\") pod \"limitador-limitador-84994f7ffc-hzcgf\" (UID: \"6aaec257-03ee-4ee4-9490-8305bf1ac755\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"
Sep 11 14:48:26.891593 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.891422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgkx8\" (UniqueName: \"kubernetes.io/projected/6aaec257-03ee-4ee4-9490-8305bf1ac755-kube-api-access-cgkx8\") pod \"limitador-limitador-84994f7ffc-hzcgf\" (UID: \"6aaec257-03ee-4ee4-9490-8305bf1ac755\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"
Sep 11 14:48:26.992300 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.992262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6aaec257-03ee-4ee4-9490-8305bf1ac755-config-file\") pod \"limitador-limitador-84994f7ffc-hzcgf\" (UID: \"6aaec257-03ee-4ee4-9490-8305bf1ac755\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"
Sep 11 14:48:26.992541 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.992367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgkx8\" (UniqueName: \"kubernetes.io/projected/6aaec257-03ee-4ee4-9490-8305bf1ac755-kube-api-access-cgkx8\") pod \"limitador-limitador-84994f7ffc-hzcgf\" (UID: \"6aaec257-03ee-4ee4-9490-8305bf1ac755\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"
Sep 11 14:48:26.992951 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:26.992931    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6aaec257-03ee-4ee4-9490-8305bf1ac755-config-file\") pod \"limitador-limitador-84994f7ffc-hzcgf\" (UID: \"6aaec257-03ee-4ee4-9490-8305bf1ac755\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"
Sep 11 14:48:27.002577 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.002551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgkx8\" (UniqueName: \"kubernetes.io/projected/6aaec257-03ee-4ee4-9490-8305bf1ac755-kube-api-access-cgkx8\") pod \"limitador-limitador-84994f7ffc-hzcgf\" (UID: \"6aaec257-03ee-4ee4-9490-8305bf1ac755\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"
Sep 11 14:48:27.090921 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.090825    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"
Sep 11 14:48:27.162308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.162279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:27.263208 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.263149    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:27.356447 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.356356    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-l9sbb"]
Sep 11 14:48:27.360063 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.360037    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l9sbb"
Sep 11 14:48:27.362666 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.362643    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vmn8r\""
Sep 11 14:48:27.366470 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.366432    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:48:27.372966 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.371342    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l9sbb"]
Sep 11 14:48:27.373865 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:48:27.373831    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aaec257_03ee_4ee4_9490_8305bf1ac755.slice/crio-287e006ae2a7e2e4c3d037deaaa170b6f372dad249105b87b96271769d702ed4 WatchSource:0}: Error finding container 287e006ae2a7e2e4c3d037deaaa170b6f372dad249105b87b96271769d702ed4: Status 404 returned error can't find the container with id 287e006ae2a7e2e4c3d037deaaa170b6f372dad249105b87b96271769d702ed4
Sep 11 14:48:27.395530 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.395492    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqsj\" (UniqueName: \"kubernetes.io/projected/242c64b0-5deb-4098-b372-86837d38c61e-kube-api-access-shqsj\") pod \"authorino-59c974bd48-l9sbb\" (UID: \"242c64b0-5deb-4098-b372-86837d38c61e\") " pod="kuadrant-system/authorino-59c974bd48-l9sbb"
Sep 11 14:48:27.495901 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.495866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqsj\" (UniqueName: \"kubernetes.io/projected/242c64b0-5deb-4098-b372-86837d38c61e-kube-api-access-shqsj\") pod \"authorino-59c974bd48-l9sbb\" (UID: \"242c64b0-5deb-4098-b372-86837d38c61e\") " pod="kuadrant-system/authorino-59c974bd48-l9sbb"
Sep 11 14:48:27.497872 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.497846    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf" event={"ID":"6aaec257-03ee-4ee4-9490-8305bf1ac755","Type":"ContainerStarted","Data":"287e006ae2a7e2e4c3d037deaaa170b6f372dad249105b87b96271769d702ed4"}
Sep 11 14:48:27.506010 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.505983    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqsj\" (UniqueName: \"kubernetes.io/projected/242c64b0-5deb-4098-b372-86837d38c61e-kube-api-access-shqsj\") pod \"authorino-59c974bd48-l9sbb\" (UID: \"242c64b0-5deb-4098-b372-86837d38c61e\") " pod="kuadrant-system/authorino-59c974bd48-l9sbb"
Sep 11 14:48:27.686356 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.686268    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l9sbb"
Sep 11 14:48:27.963208 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:27.960860    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l9sbb"]
Sep 11 14:48:27.967025 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:48:27.967000    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod242c64b0_5deb_4098_b372_86837d38c61e.slice/crio-36aabe0fbe28c8733f5be10666faede806d6ec962be87b42a9a79749cd518464 WatchSource:0}: Error finding container 36aabe0fbe28c8733f5be10666faede806d6ec962be87b42a9a79749cd518464: Status 404 returned error can't find the container with id 36aabe0fbe28c8733f5be10666faede806d6ec962be87b42a9a79749cd518464
Sep 11 14:48:28.503722 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:28.503686    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l9sbb" event={"ID":"242c64b0-5deb-4098-b372-86837d38c61e","Type":"ContainerStarted","Data":"36aabe0fbe28c8733f5be10666faede806d6ec962be87b42a9a79749cd518464"}
Sep 11 14:48:29.513801 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:29.513755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf" event={"ID":"6aaec257-03ee-4ee4-9490-8305bf1ac755","Type":"ContainerStarted","Data":"0bb496ec9eb953cbd812476b1825d7d7c4d3bcf27187864a76b71746b701d28a"}
Sep 11 14:48:29.514267 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:29.513951    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"
Sep 11 14:48:30.520010 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:30.519972    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l9sbb" event={"ID":"242c64b0-5deb-4098-b372-86837d38c61e","Type":"ContainerStarted","Data":"6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe"}
Sep 11 14:48:30.546199 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:30.546148    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf" podStartSLOduration=3.359943728 podStartE2EDuration="4.546130265s" totalImagesPullingTime="1.186186537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:48:29.533428628 +0000 UTC m=+515.856056998" watchObservedRunningTime="2026-09-11 14:48:30.546130265 +0000 UTC m=+516.868758678"
Sep 11 14:48:30.547100 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:30.547069    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-l9sbb" podStartSLOduration=1.91538347 podStartE2EDuration="3.547060417s" totalImagesPullingTime="1.631676947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:48:30.545392771 +0000 UTC m=+516.868021143" watchObservedRunningTime="2026-09-11 14:48:30.547060417 +0000 UTC m=+516.869688784"
Sep 11 14:48:40.521421 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:40.521385    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"
Sep 11 14:48:44.176114 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:44.176076    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:44.271677 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:44.271640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:44.740596 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:44.740558    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:58.437473 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:58.437411    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:58.561018 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:58.560978    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:48:58.688123 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:58.688044    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-xdnf2"]
Sep 11 14:48:58.692844 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:58.692816    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xdnf2"
Sep 11 14:48:58.697192 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:58.697165    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-xdnf2"]
Sep 11 14:48:58.733533 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:58.733497    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-xdnf2"]
Sep 11 14:48:58.733793 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:48:58.733775    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-f7xmq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-xdnf2" podUID="8cbe76d8-cbc2-4993-9ede-9f03ef128201"
Sep 11 14:48:58.773464 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:58.773416    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7xmq\" (UniqueName: \"kubernetes.io/projected/8cbe76d8-cbc2-4993-9ede-9f03ef128201-kube-api-access-f7xmq\") pod \"authorino-557f876f69-xdnf2\" (UID: \"8cbe76d8-cbc2-4993-9ede-9f03ef128201\") " pod="kuadrant-system/authorino-557f876f69-xdnf2"
Sep 11 14:48:58.874762 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:58.874716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xmq\" (UniqueName: \"kubernetes.io/projected/8cbe76d8-cbc2-4993-9ede-9f03ef128201-kube-api-access-f7xmq\") pod \"authorino-557f876f69-xdnf2\" (UID: \"8cbe76d8-cbc2-4993-9ede-9f03ef128201\") " pod="kuadrant-system/authorino-557f876f69-xdnf2"
Sep 11 14:48:58.883945 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:58.883916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7xmq\" (UniqueName: \"kubernetes.io/projected/8cbe76d8-cbc2-4993-9ede-9f03ef128201-kube-api-access-f7xmq\") pod \"authorino-557f876f69-xdnf2\" (UID: \"8cbe76d8-cbc2-4993-9ede-9f03ef128201\") " pod="kuadrant-system/authorino-557f876f69-xdnf2"
Sep 11 14:48:59.660128 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:59.660095    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xdnf2"
Sep 11 14:48:59.667568 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:59.667545    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xdnf2"
Sep 11 14:48:59.780926 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:59.780888    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7xmq\" (UniqueName: \"kubernetes.io/projected/8cbe76d8-cbc2-4993-9ede-9f03ef128201-kube-api-access-f7xmq\") pod \"8cbe76d8-cbc2-4993-9ede-9f03ef128201\" (UID: \"8cbe76d8-cbc2-4993-9ede-9f03ef128201\") "
Sep 11 14:48:59.783025 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:59.782994    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbe76d8-cbc2-4993-9ede-9f03ef128201-kube-api-access-f7xmq" (OuterVolumeSpecName: "kube-api-access-f7xmq") pod "8cbe76d8-cbc2-4993-9ede-9f03ef128201" (UID: "8cbe76d8-cbc2-4993-9ede-9f03ef128201"). InnerVolumeSpecName "kube-api-access-f7xmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:48:59.881599 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:48:59.881559    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7xmq\" (UniqueName: \"kubernetes.io/projected/8cbe76d8-cbc2-4993-9ede-9f03ef128201-kube-api-access-f7xmq\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:49:00.664683 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:00.664651    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xdnf2"
Sep 11 14:49:00.696177 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:00.696139    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-xdnf2"]
Sep 11 14:49:00.702734 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:00.702702    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-xdnf2"]
Sep 11 14:49:00.754576 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:00.754541    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l9sbb"]
Sep 11 14:49:00.754797 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:00.754752    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-l9sbb" podUID="242c64b0-5deb-4098-b372-86837d38c61e" containerName="authorino" containerID="cri-o://6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe" gracePeriod=30
Sep 11 14:49:00.998706 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:00.998678    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l9sbb"
Sep 11 14:49:01.093497 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.093446    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shqsj\" (UniqueName: \"kubernetes.io/projected/242c64b0-5deb-4098-b372-86837d38c61e-kube-api-access-shqsj\") pod \"242c64b0-5deb-4098-b372-86837d38c61e\" (UID: \"242c64b0-5deb-4098-b372-86837d38c61e\") "
Sep 11 14:49:01.095384 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.095348    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242c64b0-5deb-4098-b372-86837d38c61e-kube-api-access-shqsj" (OuterVolumeSpecName: "kube-api-access-shqsj") pod "242c64b0-5deb-4098-b372-86837d38c61e" (UID: "242c64b0-5deb-4098-b372-86837d38c61e"). InnerVolumeSpecName "kube-api-access-shqsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:49:01.195027 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.194939    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqsj\" (UniqueName: \"kubernetes.io/projected/242c64b0-5deb-4098-b372-86837d38c61e-kube-api-access-shqsj\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:49:01.670640 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.670608    2559 generic.go:356] "Generic (PLEG): container finished" podID="242c64b0-5deb-4098-b372-86837d38c61e" containerID="6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe" exitCode=0
Sep 11 14:49:01.671116 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.670647    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l9sbb"
Sep 11 14:49:01.671116 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.670695    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l9sbb" event={"ID":"242c64b0-5deb-4098-b372-86837d38c61e","Type":"ContainerDied","Data":"6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe"}
Sep 11 14:49:01.671116 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.670743    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l9sbb" event={"ID":"242c64b0-5deb-4098-b372-86837d38c61e","Type":"ContainerDied","Data":"36aabe0fbe28c8733f5be10666faede806d6ec962be87b42a9a79749cd518464"}
Sep 11 14:49:01.671116 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.670768    2559 scope.go:117] "RemoveContainer" containerID="6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe"
Sep 11 14:49:01.679314 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.679297    2559 scope.go:117] "RemoveContainer" containerID="6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe"
Sep 11 14:49:01.679636 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:49:01.679617    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe\": container with ID starting with 6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe not found: ID does not exist" containerID="6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe"
Sep 11 14:49:01.679689 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.679646    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe"} err="failed to get container status \"6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe\": rpc error: code = NotFound desc = could not find container \"6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe\": container with ID starting with 6a89837a8da615a9dc8574ae7776d380ae4250c9ac1f84c2c7f7e5113734e8fe not found: ID does not exist"
Sep 11 14:49:01.691867 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.691839    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l9sbb"]
Sep 11 14:49:01.695866 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:01.695843    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l9sbb"]
Sep 11 14:49:02.179528 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:02.179496    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="242c64b0-5deb-4098-b372-86837d38c61e" path="/var/lib/kubelet/pods/242c64b0-5deb-4098-b372-86837d38c61e/volumes"
Sep 11 14:49:02.179822 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:02.179809    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cbe76d8-cbc2-4993-9ede-9f03ef128201" path="/var/lib/kubelet/pods/8cbe76d8-cbc2-4993-9ede-9f03ef128201/volumes"
Sep 11 14:49:05.940283 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:05.940243    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:06.043860 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:06.043827    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:06.140279 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:06.140247    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:06.225293 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:06.225254    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:06.302352 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:06.302304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:09.515019 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:09.514984    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:09.612178 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:09.612135    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:35.535088 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:35.535051    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:43.155637 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:43.155597    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:43.244488 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:43.244431    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:44.700781 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:44.700748    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:44.805060 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:44.805021    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:44.900154 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:44.900103    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:45.017377 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:45.017339    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:45.251442 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:45.251412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:45.384432 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:45.384323    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:45.611302 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:45.611269    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:49:55.414577 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:55.414538    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:49:55.513852 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:49:55.513814    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:50:25.985874 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:25.985792    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:50:26.350789 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:26.350699    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:50:35.710849 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.710816    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"]
Sep 11 14:50:35.711234 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.711144    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242c64b0-5deb-4098-b372-86837d38c61e" containerName="authorino"
Sep 11 14:50:35.711234 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.711155    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="242c64b0-5deb-4098-b372-86837d38c61e" containerName="authorino"
Sep 11 14:50:35.711234 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.711226    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="242c64b0-5deb-4098-b372-86837d38c61e" containerName="authorino"
Sep 11 14:50:35.714308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.714291    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.719902 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.719878    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 14:50:35.720039 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.719916    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 14:50:35.720039 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.719971    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 14:50:35.720039 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.719916    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cp687\""
Sep 11 14:50:35.732641 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.732601    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"]
Sep 11 14:50:35.761358 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.761321    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:50:35.827520 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.827476    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.827704 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.827546    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.827704 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.827628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.827704 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.827663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7twc\" (UniqueName: \"kubernetes.io/projected/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-kube-api-access-k7twc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.827704 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.827697    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.827839 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.827731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.928842 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.928796    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.929012 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.928851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.929012 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.928871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7twc\" (UniqueName: \"kubernetes.io/projected/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-kube-api-access-k7twc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.929012 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.928902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.929012 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.928922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.929012 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.928954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.929297 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.929270    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.929354 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.929320    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.929392 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.929355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.931119 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.931096    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.931372 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.931355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:35.938994 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:35.938963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7twc\" (UniqueName: \"kubernetes.io/projected/f8e3d64b-3965-4d96-ba52-6485f4d1ef9d-kube-api-access-k7twc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-njbrp\" (UID: \"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:36.024919 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:36.024879    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:36.299404 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:36.299379    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"]
Sep 11 14:50:36.302042 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:50:36.302012    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e3d64b_3965_4d96_ba52_6485f4d1ef9d.slice/crio-67d574e6e8a85af03393f6316d5710478281bdaa621cce1c2774ef0d1d4ad236 WatchSource:0}: Error finding container 67d574e6e8a85af03393f6316d5710478281bdaa621cce1c2774ef0d1d4ad236: Status 404 returned error can't find the container with id 67d574e6e8a85af03393f6316d5710478281bdaa621cce1c2774ef0d1d4ad236
Sep 11 14:50:36.303345 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:36.303324    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:50:36.413946 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:36.413904    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:50:37.105017 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:37.104975    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp" event={"ID":"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d","Type":"ContainerStarted","Data":"67d574e6e8a85af03393f6316d5710478281bdaa621cce1c2774ef0d1d4ad236"}
Sep 11 14:50:38.111400 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:38.111366    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp" event={"ID":"f8e3d64b-3965-4d96-ba52-6485f4d1ef9d","Type":"ContainerStarted","Data":"cc42e79f313baf0488cd9770cd0e5c732048d463266c46f549897410a3613929"}
Sep 11 14:50:38.132956 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:38.132897    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp" podStartSLOduration=2.08908289 podStartE2EDuration="3.13287995s" totalImagesPullingTime="1.04379706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:50:38.131951659 +0000 UTC m=+644.454580028" watchObservedRunningTime="2026-09-11 14:50:38.13287995 +0000 UTC m=+644.455508320"
Sep 11 14:50:46.025046 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:46.025009    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:46.025481 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:46.025080    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:46.037145 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:46.037113    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:46.159478 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:46.159425    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-njbrp"
Sep 11 14:50:55.715877 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:55.715841    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:50:56.626337 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:50:56.626300    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:51:04.364470 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:04.364426    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:51:04.465182 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:04.465145    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:51:07.281838 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.281803    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"]
Sep 11 14:51:07.288084 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.288062    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.290881 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.290861    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 14:51:07.301469 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.297519    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"]
Sep 11 14:51:07.304127 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.304100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:51:07.403950 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.403923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.403950 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.403958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.404158 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.403974    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.404158 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.404102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ab935e-838e-4597-ac0b-276d522ddbc8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.404266 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.404164    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.404266 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.404214    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9w9f\" (UniqueName: \"kubernetes.io/projected/62ab935e-838e-4597-ac0b-276d522ddbc8-kube-api-access-q9w9f\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.504861 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.504819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ab935e-838e-4597-ac0b-276d522ddbc8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.505057 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.504871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.505057 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.505004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9w9f\" (UniqueName: \"kubernetes.io/projected/62ab935e-838e-4597-ac0b-276d522ddbc8-kube-api-access-q9w9f\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.505182 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.505060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.505182 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.505091    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.505182 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.505106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.505511 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.505449    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.505638 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.505554    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.505638 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.505567    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.507172 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.507142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ab935e-838e-4597-ac0b-276d522ddbc8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.507395 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.507380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ab935e-838e-4597-ac0b-276d522ddbc8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.514154 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.514126    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9w9f\" (UniqueName: \"kubernetes.io/projected/62ab935e-838e-4597-ac0b-276d522ddbc8-kube-api-access-q9w9f\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2wtcx\" (UID: \"62ab935e-838e-4597-ac0b-276d522ddbc8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.599356 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.599262    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:07.871690 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:07.871658    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"]
Sep 11 14:51:07.873895 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:51:07.873867    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ab935e_838e_4597_ac0b_276d522ddbc8.slice/crio-366380b07dd23eca99082ad8f4fa43b82725fa76a4cc318afc42e182cdc016e4 WatchSource:0}: Error finding container 366380b07dd23eca99082ad8f4fa43b82725fa76a4cc318afc42e182cdc016e4: Status 404 returned error can't find the container with id 366380b07dd23eca99082ad8f4fa43b82725fa76a4cc318afc42e182cdc016e4
Sep 11 14:51:08.247886 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:08.247848    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx" event={"ID":"62ab935e-838e-4597-ac0b-276d522ddbc8","Type":"ContainerStarted","Data":"0fbf94497482798abde78e3c4e8f60711309eddd9f5723ff36e986097dc6e3b7"}
Sep 11 14:51:08.248045 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:08.247913    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx" event={"ID":"62ab935e-838e-4597-ac0b-276d522ddbc8","Type":"ContainerStarted","Data":"366380b07dd23eca99082ad8f4fa43b82725fa76a4cc318afc42e182cdc016e4"}
Sep 11 14:51:08.267710 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:08.267650    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx" podStartSLOduration=1.056292729 podStartE2EDuration="1.267631666s" totalImagesPullingTime="211.338937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:51:08.266434522 +0000 UTC m=+674.589062890" watchObservedRunningTime="2026-09-11 14:51:08.267631666 +0000 UTC m=+674.590260034"
Sep 11 14:51:08.616465 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:08.616426    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:51:16.691777 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:16.691738    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:51:16.751596 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:16.751564    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:51:17.599866 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:17.599833    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:17.600126 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:17.600105    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:17.612140 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:17.612115    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:18.305575 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:18.305545    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2wtcx"
Sep 11 14:51:29.505927 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:29.505883    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:51:30.922216 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:30.922176    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:51:45.009592 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:45.009540    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:51:46.823387 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:46.823304    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:51:53.775634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.775603    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"]
Sep 11 14:51:53.778924 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.778903    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.782003 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:51:53.781972    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="e147bc5d-67c5-4713-8904-b7dcd06f3563" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.782199 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:51:53.782179    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"premium-simulated-simulated-premium-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 11 14:51:53.793991 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.793966    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"]
Sep 11 14:51:53.807368 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.807339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.807506 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.807391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e147bc5d-67c5-4713-8904-b7dcd06f3563-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.807506 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.807446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.807602 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.807506    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.807602 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.807576    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wqvc\" (UniqueName: \"kubernetes.io/projected/e147bc5d-67c5-4713-8904-b7dcd06f3563-kube-api-access-2wqvc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.807602 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.807598    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.823723 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.823698    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:51:53.908297 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.908269    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e147bc5d-67c5-4713-8904-b7dcd06f3563-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.908419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.908304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.908419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.908322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.908419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.908368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wqvc\" (UniqueName: \"kubernetes.io/projected/e147bc5d-67c5-4713-8904-b7dcd06f3563-kube-api-access-2wqvc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.908419 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.908389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.908634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.908505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.908704 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.908675    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.908768 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.908734    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.908768 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.908751    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.910563 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.910543    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e147bc5d-67c5-4713-8904-b7dcd06f3563-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:53.928091 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:53.928065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wqvc\" (UniqueName: \"kubernetes.io/projected/e147bc5d-67c5-4713-8904-b7dcd06f3563-kube-api-access-2wqvc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:54.909438 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:51:54.909395    2559 secret.go:189] Couldn't get secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition
Sep 11 14:51:54.909815 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:51:54.909501    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e147bc5d-67c5-4713-8904-b7dcd06f3563-tls-certs podName:e147bc5d-67c5-4713-8904-b7dcd06f3563 nodeName:}" failed. No retries permitted until 2026-09-11 14:51:55.409485073 +0000 UTC m=+721.732113433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e147bc5d-67c5-4713-8904-b7dcd06f3563-tls-certs") pod "premium-simulated-simulated-premium-kserve-57fc66f697-xtz29" (UID: "e147bc5d-67c5-4713-8904-b7dcd06f3563") : failed to sync secret cache: timed out waiting for the condition
Sep 11 14:51:55.000950 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:55.000916    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 14:51:55.421807 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:55.421771    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e147bc5d-67c5-4713-8904-b7dcd06f3563-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:55.424080 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:55.424060    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e147bc5d-67c5-4713-8904-b7dcd06f3563-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xtz29\" (UID: \"e147bc5d-67c5-4713-8904-b7dcd06f3563\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:55.601367 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:55.601334    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:51:55.875633 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:55.875607    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"]
Sep 11 14:51:55.877388 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:51:55.877361    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode147bc5d_67c5_4713_8904_b7dcd06f3563.slice/crio-63d83fed786cb5b05a3db57f0df376070a89d0bc0a5074159e4d4df193e536c2 WatchSource:0}: Error finding container 63d83fed786cb5b05a3db57f0df376070a89d0bc0a5074159e4d4df193e536c2: Status 404 returned error can't find the container with id 63d83fed786cb5b05a3db57f0df376070a89d0bc0a5074159e4d4df193e536c2
Sep 11 14:51:56.056898 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:56.056866    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:51:56.468654 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:56.468562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29" event={"ID":"e147bc5d-67c5-4713-8904-b7dcd06f3563","Type":"ContainerStarted","Data":"9ec40b41dcf9dab924357783a9f48a8afcb2c4bfcedafde725b8960111542b6d"}
Sep 11 14:51:56.468654 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:51:56.468601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29" event={"ID":"e147bc5d-67c5-4713-8904-b7dcd06f3563","Type":"ContainerStarted","Data":"63d83fed786cb5b05a3db57f0df376070a89d0bc0a5074159e4d4df193e536c2"}
Sep 11 14:52:03.534313 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:03.534273    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:52:05.601962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:05.601924    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:52:05.602345 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:05.601979    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:52:05.614234 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:05.614209    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:52:05.635563 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:05.635521    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29" podStartSLOduration=12.43361589 podStartE2EDuration="12.635507668s" totalImagesPullingTime="201.891778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:51:56.510584168 +0000 UTC m=+722.833212536" watchObservedRunningTime="2026-09-11 14:52:05.635507668 +0000 UTC m=+731.958136037"
Sep 11 14:52:06.522552 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:06.522520    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xtz29"
Sep 11 14:52:39.469558 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:39.469524    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:52:39.534464 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:39.534411    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:52:51.549389 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.549353    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-948bd5744-xs867"]
Sep 11 14:52:51.552889 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.552865    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-948bd5744-xs867"
Sep 11 14:52:51.556716 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.556692    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 14:52:51.556819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.556726    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vmn8r\""
Sep 11 14:52:51.563615 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.563596    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-948bd5744-xs867"]
Sep 11 14:52:51.622072 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.622045    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjp2s\" (UniqueName: \"kubernetes.io/projected/dba830e1-6091-4587-b619-92268c5bac62-kube-api-access-cjp2s\") pod \"authorino-948bd5744-xs867\" (UID: \"dba830e1-6091-4587-b619-92268c5bac62\") " pod="kuadrant-system/authorino-948bd5744-xs867"
Sep 11 14:52:51.622202 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.622077    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dba830e1-6091-4587-b619-92268c5bac62-tls-cert\") pod \"authorino-948bd5744-xs867\" (UID: \"dba830e1-6091-4587-b619-92268c5bac62\") " pod="kuadrant-system/authorino-948bd5744-xs867"
Sep 11 14:52:51.723236 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.723207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjp2s\" (UniqueName: \"kubernetes.io/projected/dba830e1-6091-4587-b619-92268c5bac62-kube-api-access-cjp2s\") pod \"authorino-948bd5744-xs867\" (UID: \"dba830e1-6091-4587-b619-92268c5bac62\") " pod="kuadrant-system/authorino-948bd5744-xs867"
Sep 11 14:52:51.723236 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.723238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dba830e1-6091-4587-b619-92268c5bac62-tls-cert\") pod \"authorino-948bd5744-xs867\" (UID: \"dba830e1-6091-4587-b619-92268c5bac62\") " pod="kuadrant-system/authorino-948bd5744-xs867"
Sep 11 14:52:51.725567 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.725543    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dba830e1-6091-4587-b619-92268c5bac62-tls-cert\") pod \"authorino-948bd5744-xs867\" (UID: \"dba830e1-6091-4587-b619-92268c5bac62\") " pod="kuadrant-system/authorino-948bd5744-xs867"
Sep 11 14:52:51.733232 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.733206    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjp2s\" (UniqueName: \"kubernetes.io/projected/dba830e1-6091-4587-b619-92268c5bac62-kube-api-access-cjp2s\") pod \"authorino-948bd5744-xs867\" (UID: \"dba830e1-6091-4587-b619-92268c5bac62\") " pod="kuadrant-system/authorino-948bd5744-xs867"
Sep 11 14:52:51.861882 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:51.861820    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-948bd5744-xs867"
Sep 11 14:52:52.130216 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:52.130192    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-948bd5744-xs867"]
Sep 11 14:52:52.132136 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:52:52.132106    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddba830e1_6091_4587_b619_92268c5bac62.slice/crio-00c16f14dfb8cef1911d9b5eee0b5298c3132e6dd784da417bde8e12bd1704ae WatchSource:0}: Error finding container 00c16f14dfb8cef1911d9b5eee0b5298c3132e6dd784da417bde8e12bd1704ae: Status 404 returned error can't find the container with id 00c16f14dfb8cef1911d9b5eee0b5298c3132e6dd784da417bde8e12bd1704ae
Sep 11 14:52:52.718308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:52.718276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-948bd5744-xs867" event={"ID":"dba830e1-6091-4587-b619-92268c5bac62","Type":"ContainerStarted","Data":"00c16f14dfb8cef1911d9b5eee0b5298c3132e6dd784da417bde8e12bd1704ae"}
Sep 11 14:52:53.725514 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:53.725474    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-948bd5744-xs867" event={"ID":"dba830e1-6091-4587-b619-92268c5bac62","Type":"ContainerStarted","Data":"92cd78ad980e935bb3b64652892a0f839ba52655945f8b489949f10babdf5cd9"}
Sep 11 14:52:53.746418 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:53.746358    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-948bd5744-xs867" podStartSLOduration=2.056190707 podStartE2EDuration="2.746342686s" totalImagesPullingTime="690.151979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:52:53.745227388 +0000 UTC m=+780.067855756" watchObservedRunningTime="2026-09-11 14:52:53.746342686 +0000 UTC m=+780.068971055"
Sep 11 14:52:54.469821 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:54.469777    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:52:54.572315 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:54.572283    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:52:55.883095 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:55.883052    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:52:55.980039 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:52:55.980007    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:14.542416 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:14.542375    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:14.646163 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:14.646129    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:14.943297 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:14.943193    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:15.039656 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:15.039623    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:15.145134 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:15.145097    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:15.330401 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:15.330361    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:16.138703 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:16.138662    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:16.240560 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:16.240522    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:16.680772 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:16.680717    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:16.843547 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:16.843511    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7683-8442ad"]
Sep 11 14:53:16.847016 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:16.846999    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-8442ad"
Sep 11 14:53:16.849762 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:16.849735    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5lj77\""
Sep 11 14:53:16.851965 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:16.851939    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7683-8442ad"]
Sep 11 14:53:16.942135 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:16.942051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dllbm\" (UniqueName: \"kubernetes.io/projected/b37116eb-db04-4d34-9d95-b7f24a7305be-kube-api-access-dllbm\") pod \"test-curl-7683-8442ad\" (UID: \"b37116eb-db04-4d34-9d95-b7f24a7305be\") " pod="opendatahub/test-curl-7683-8442ad"
Sep 11 14:53:17.042648 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:17.042609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dllbm\" (UniqueName: \"kubernetes.io/projected/b37116eb-db04-4d34-9d95-b7f24a7305be-kube-api-access-dllbm\") pod \"test-curl-7683-8442ad\" (UID: \"b37116eb-db04-4d34-9d95-b7f24a7305be\") " pod="opendatahub/test-curl-7683-8442ad"
Sep 11 14:53:17.051736 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:17.051703    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dllbm\" (UniqueName: \"kubernetes.io/projected/b37116eb-db04-4d34-9d95-b7f24a7305be-kube-api-access-dllbm\") pod \"test-curl-7683-8442ad\" (UID: \"b37116eb-db04-4d34-9d95-b7f24a7305be\") " pod="opendatahub/test-curl-7683-8442ad"
Sep 11 14:53:17.157281 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:17.157248    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-8442ad"
Sep 11 14:53:17.426578 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:17.426481    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7683-8442ad"]
Sep 11 14:53:17.428997 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:53:17.428963    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb37116eb_db04_4d34_9d95_b7f24a7305be.slice/crio-829b047201f0cc4cdcc7de41cacc276579d584e231f661e21a1ea6b6ed86decd WatchSource:0}: Error finding container 829b047201f0cc4cdcc7de41cacc276579d584e231f661e21a1ea6b6ed86decd: Status 404 returned error can't find the container with id 829b047201f0cc4cdcc7de41cacc276579d584e231f661e21a1ea6b6ed86decd
Sep 11 14:53:17.842878 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:17.842842    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7683-8442ad" event={"ID":"b37116eb-db04-4d34-9d95-b7f24a7305be","Type":"ContainerStarted","Data":"829b047201f0cc4cdcc7de41cacc276579d584e231f661e21a1ea6b6ed86decd"}
Sep 11 14:53:18.133617 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:18.133538    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:53:18.165976 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:18.165941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:18.267402 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:18.267366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:18.376839 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:18.376801    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:18.465893 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:18.465792    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:19.441446 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:19.441413    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:19.541236 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:19.541196    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:19.673065 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:19.673023    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:19.855020 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:19.854985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7683-8442ad" event={"ID":"b37116eb-db04-4d34-9d95-b7f24a7305be","Type":"ContainerStarted","Data":"64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0"}
Sep 11 14:53:19.874029 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:19.873979    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7683-8442ad" podStartSLOduration=1.682548094 podStartE2EDuration="3.873963553s" totalImagesPullingTime="2.191415459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:19.872501101 +0000 UTC m=+806.195129472" watchObservedRunningTime="2026-09-11 14:53:19.873963553 +0000 UTC m=+806.196591921"
Sep 11 14:53:20.087601 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.087572    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:20.187491 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.187393    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:20.532252 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.532217    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7683-8442ad"]
Sep 11 14:53:20.537057 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.537033    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7683-8442ad"]
Sep 11 14:53:20.706895 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.706860    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7683-1f9f3a"]
Sep 11 14:53:20.707235 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.707222    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37116eb-db04-4d34-9d95-b7f24a7305be" containerName="test-curl-7683-8442ad"
Sep 11 14:53:20.707299 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.707237    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37116eb-db04-4d34-9d95-b7f24a7305be" containerName="test-curl-7683-8442ad"
Sep 11 14:53:20.707344 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.707328    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b37116eb-db04-4d34-9d95-b7f24a7305be" containerName="test-curl-7683-8442ad"
Sep 11 14:53:20.710824 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.710801    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-1f9f3a"
Sep 11 14:53:20.720666 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.720637    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7683-1f9f3a"]
Sep 11 14:53:20.773342 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.773302    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkfw\" (UniqueName: \"kubernetes.io/projected/ef67cf38-57a2-4229-8c15-c1139867d634-kube-api-access-jvkfw\") pod \"test-curl-7683-1f9f3a\" (UID: \"ef67cf38-57a2-4229-8c15-c1139867d634\") " pod="opendatahub/test-curl-7683-1f9f3a"
Sep 11 14:53:20.860117 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:20.860028    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7683-8442ad" container="test-curl-7683-8442ad"
Sep 11 14:53:20.874387 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.874358    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkfw\" (UniqueName: \"kubernetes.io/projected/ef67cf38-57a2-4229-8c15-c1139867d634-kube-api-access-jvkfw\") pod \"test-curl-7683-1f9f3a\" (UID: \"ef67cf38-57a2-4229-8c15-c1139867d634\") " pod="opendatahub/test-curl-7683-1f9f3a"
Sep 11 14:53:20.901595 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.901563    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkfw\" (UniqueName: \"kubernetes.io/projected/ef67cf38-57a2-4229-8c15-c1139867d634-kube-api-access-jvkfw\") pod \"test-curl-7683-1f9f3a\" (UID: \"ef67cf38-57a2-4229-8c15-c1139867d634\") " pod="opendatahub/test-curl-7683-1f9f3a"
Sep 11 14:53:20.965961 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:20.965921    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:21.025169 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:21.025127    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-1f9f3a"
Sep 11 14:53:21.063977 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:21.063942    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:21.169755 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:21.169659    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:21.302383 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:21.302351    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7683-1f9f3a"]
Sep 11 14:53:21.303618 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:53:21.303591    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef67cf38_57a2_4229_8c15_c1139867d634.slice/crio-9de1df6a47140a7849c19559e69d2ba1c24c569746ae814384e40eed9f1529cd WatchSource:0}: Error finding container 9de1df6a47140a7849c19559e69d2ba1c24c569746ae814384e40eed9f1529cd: Status 404 returned error can't find the container with id 9de1df6a47140a7849c19559e69d2ba1c24c569746ae814384e40eed9f1529cd
Sep 11 14:53:21.527029 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:21.526996    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:21.662434 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:21.662389    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:21.845405 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:21.845327    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:21.867848 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:21.867814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7683-1f9f3a" event={"ID":"ef67cf38-57a2-4229-8c15-c1139867d634","Type":"ContainerStarted","Data":"caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf"}
Sep 11 14:53:21.868105 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:21.867860    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7683-1f9f3a" event={"ID":"ef67cf38-57a2-4229-8c15-c1139867d634","Type":"ContainerStarted","Data":"9de1df6a47140a7849c19559e69d2ba1c24c569746ae814384e40eed9f1529cd"}
Sep 11 14:53:21.868105 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:21.867967    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7683-8442ad" podUID="b37116eb-db04-4d34-9d95-b7f24a7305be" containerName="test-curl-7683-8442ad" containerID="cri-o://64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0" gracePeriod=2
Sep 11 14:53:21.899505 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:21.899433    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7683-1f9f3a" podStartSLOduration=1.8994168299999998 podStartE2EDuration="1.89941683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:21.897930113 +0000 UTC m=+808.220558484" watchObservedRunningTime="2026-09-11 14:53:21.89941683 +0000 UTC m=+808.222045201"
Sep 11 14:53:21.900443 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:21.900402    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7683-8442ad\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="b37116eb-db04-4d34-9d95-b7f24a7305be" pod="opendatahub/test-curl-7683-8442ad"
Sep 11 14:53:22.429860 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:22.429816    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:22.585307 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:22.585227    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7683-1f9f3a"]
Sep 11 14:53:22.595442 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:22.595411    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7683-1f9f3a"]
Sep 11 14:53:22.872905 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:22.872824    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7683-1f9f3a" container="test-curl-7683-1f9f3a"
Sep 11 14:53:23.222123 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:23.222086    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:23.384318 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:23.384275    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:23.593219 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:23.593117    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:23.878146 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:23.878113    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7683-1f9f3a" podUID="ef67cf38-57a2-4229-8c15-c1139867d634" containerName="test-curl-7683-1f9f3a" containerID="cri-o://caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf" gracePeriod=2
Sep 11 14:53:23.881161 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:23.881133    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7683-1f9f3a\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="ef67cf38-57a2-4229-8c15-c1139867d634" pod="opendatahub/test-curl-7683-1f9f3a"
Sep 11 14:53:23.985819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:23.985786    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:24.019468 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.019436    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-8442ad"
Sep 11 14:53:24.022825 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:24.022797    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7683-1f9f3a\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="ef67cf38-57a2-4229-8c15-c1139867d634" pod="opendatahub/test-curl-7683-1f9f3a"
Sep 11 14:53:24.103158 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.103114    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dllbm\" (UniqueName: \"kubernetes.io/projected/b37116eb-db04-4d34-9d95-b7f24a7305be-kube-api-access-dllbm\") pod \"b37116eb-db04-4d34-9d95-b7f24a7305be\" (UID: \"b37116eb-db04-4d34-9d95-b7f24a7305be\") "
Sep 11 14:53:24.105467 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.105415    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37116eb-db04-4d34-9d95-b7f24a7305be-kube-api-access-dllbm" (OuterVolumeSpecName: "kube-api-access-dllbm") pod "b37116eb-db04-4d34-9d95-b7f24a7305be" (UID: "b37116eb-db04-4d34-9d95-b7f24a7305be"). InnerVolumeSpecName "kube-api-access-dllbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:53:24.178677 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:24.178578    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7683-1f9f3a\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="ef67cf38-57a2-4229-8c15-c1139867d634" pod="opendatahub/test-curl-7683-1f9f3a"
Sep 11 14:53:24.180155 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.180122    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b37116eb-db04-4d34-9d95-b7f24a7305be" path="/var/lib/kubelet/pods/b37116eb-db04-4d34-9d95-b7f24a7305be/volumes"
Sep 11 14:53:24.204477 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.204403    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dllbm\" (UniqueName: \"kubernetes.io/projected/b37116eb-db04-4d34-9d95-b7f24a7305be-kube-api-access-dllbm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:53:24.484688 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.484653    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:24.669906 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.669871    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:24.769910 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.769808    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:24.884742 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.884702    2559 generic.go:356] "Generic (PLEG): container finished" podID="b37116eb-db04-4d34-9d95-b7f24a7305be" containerID="64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0" exitCode=137
Sep 11 14:53:24.885323 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.885303    2559 scope.go:117] "RemoveContainer" containerID="64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0"
Sep 11 14:53:24.885640 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.885620    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-8442ad"
Sep 11 14:53:24.899156 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.899121    2559 scope.go:117] "RemoveContainer" containerID="64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0"
Sep 11 14:53:24.899576 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:24.899552    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0\": container with ID starting with 64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0 not found: ID does not exist" containerID="64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0"
Sep 11 14:53:24.899644 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.899585    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0"} err="failed to get container status \"64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0\": rpc error: code = NotFound desc = could not find container \"64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0\": container with ID starting with 64914fc4a88f97dca57821f2b756ed5e73d657792a3523da17017d7f118317e0 not found: ID does not exist"
Sep 11 14:53:24.949968 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:24.949928    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:25.257590 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.257550    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"]
Sep 11 14:53:25.258289 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.258258    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef67cf38-57a2-4229-8c15-c1139867d634" containerName="test-curl-7683-1f9f3a"
Sep 11 14:53:25.258289 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.258288    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef67cf38-57a2-4229-8c15-c1139867d634" containerName="test-curl-7683-1f9f3a"
Sep 11 14:53:25.258503 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.258411    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef67cf38-57a2-4229-8c15-c1139867d634" containerName="test-curl-7683-1f9f3a"
Sep 11 14:53:25.263518 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.263491    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.266423 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.266400    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f722bf59-openshift-default-dockercfg-d5wsp\""
Sep 11 14:53:25.266564 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.266428    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 14:53:25.278837 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.278808    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"]
Sep 11 14:53:25.308207 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.308170    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:25.314332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.314300    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-credential-socket\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.314332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.314338    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-socket\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.314581 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.314362    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgxfb\" (UniqueName: \"kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-kube-api-access-bgxfb\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.314643 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.314574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c163109-e97d-438a-93f9-85efa4d0e776-istiod-ca-cert\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.314701 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.314650    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c163109-e97d-438a-93f9-85efa4d0e776-istio-podinfo\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.314701 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.314679    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-istio-token\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.314795 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.314751    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-envoy\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.314846 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.314801    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-data\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.314846 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.314828    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-certs\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.411294 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.411254    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:25.415809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.415775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c163109-e97d-438a-93f9-85efa4d0e776-istio-podinfo\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.415809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.415810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-istio-token\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416017 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.415851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-envoy\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416017 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.415873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-data\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416017 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.415896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-certs\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416186 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.416052    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-credential-socket\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416186 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.416108    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-socket\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416186 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.416131    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgxfb\" (UniqueName: \"kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-kube-api-access-bgxfb\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416352 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.416254    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c163109-e97d-438a-93f9-85efa4d0e776-istiod-ca-cert\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416352 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.416301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-data\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416512 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.416385    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-certs\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416512 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.416471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-credential-socket\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416810 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.416786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-socket\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.416979 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.416955    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c163109-e97d-438a-93f9-85efa4d0e776-istiod-ca-cert\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.418825 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.418801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-envoy\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.418931 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.418913    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c163109-e97d-438a-93f9-85efa4d0e776-istio-podinfo\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.425514 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.425485    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-istio-token\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.428143 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.428120    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgxfb\" (UniqueName: \"kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-kube-api-access-bgxfb\") pod \"e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") " pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.575129 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.575048    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:25.748713 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.748665    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:25.866180 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.866146    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"]
Sep 11 14:53:25.869443 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:53:25.869404    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c163109_e97d_438a_93f9_85efa4d0e776.slice/crio-b0b1ac2764bcedbe6ee302d8dba0ddc5643b28226d90ea21979256e96b1d1f6d WatchSource:0}: Error finding container b0b1ac2764bcedbe6ee302d8dba0ddc5643b28226d90ea21979256e96b1d1f6d: Status 404 returned error can't find the container with id b0b1ac2764bcedbe6ee302d8dba0ddc5643b28226d90ea21979256e96b1d1f6d
Sep 11 14:53:25.897279 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:25.897171    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" event={"ID":"6c163109-e97d-438a-93f9-85efa4d0e776","Type":"ContainerStarted","Data":"b0b1ac2764bcedbe6ee302d8dba0ddc5643b28226d90ea21979256e96b1d1f6d"}
Sep 11 14:53:26.081129 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.081094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:26.290036 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.289992    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:26.505536 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.505511    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-1f9f3a"
Sep 11 14:53:26.527499 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.527466    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvkfw\" (UniqueName: \"kubernetes.io/projected/ef67cf38-57a2-4229-8c15-c1139867d634-kube-api-access-jvkfw\") pod \"ef67cf38-57a2-4229-8c15-c1139867d634\" (UID: \"ef67cf38-57a2-4229-8c15-c1139867d634\") "
Sep 11 14:53:26.529483 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.529444    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef67cf38-57a2-4229-8c15-c1139867d634-kube-api-access-jvkfw" (OuterVolumeSpecName: "kube-api-access-jvkfw") pod "ef67cf38-57a2-4229-8c15-c1139867d634" (UID: "ef67cf38-57a2-4229-8c15-c1139867d634"). InnerVolumeSpecName "kube-api-access-jvkfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:53:26.628533 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.628430    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvkfw\" (UniqueName: \"kubernetes.io/projected/ef67cf38-57a2-4229-8c15-c1139867d634-kube-api-access-jvkfw\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:53:26.847152 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.847118    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:26.904508 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.904392    2559 generic.go:356] "Generic (PLEG): container finished" podID="ef67cf38-57a2-4229-8c15-c1139867d634" containerID="caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf" exitCode=137
Sep 11 14:53:26.904508 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.904477    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-1f9f3a"
Sep 11 14:53:26.905073 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.904483    2559 scope.go:117] "RemoveContainer" containerID="caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf"
Sep 11 14:53:26.914536 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.914513    2559 scope.go:117] "RemoveContainer" containerID="caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf"
Sep 11 14:53:26.914887 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:26.914864    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf\": container with ID starting with caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf not found: ID does not exist" containerID="caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf"
Sep 11 14:53:26.914961 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.914900    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf"} err="failed to get container status \"caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf\": rpc error: code = NotFound desc = could not find container \"caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf\": container with ID starting with caf94cb208fd494df2c1493264b38a680cb5bf94a56a2965d11b0da04c2e1faf not found: ID does not exist"
Sep 11 14:53:26.916499 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:26.916395    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7683-1f9f3a\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="ef67cf38-57a2-4229-8c15-c1139867d634" pod="opendatahub/test-curl-7683-1f9f3a"
Sep 11 14:53:26.940321 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:26.940278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:27.568681 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:27.568627    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:27.703800 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:27.703762    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:27.917138 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:27.917049    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:28.182104 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:28.182019    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef67cf38-57a2-4229-8c15-c1139867d634" path="/var/lib/kubelet/pods/ef67cf38-57a2-4229-8c15-c1139867d634/volumes"
Sep 11 14:53:28.522614 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:28.522576    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:53:28.522682 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:28.522667    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:53:28.522718 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:28.522696    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:53:28.919050 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:28.918962    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" event={"ID":"6c163109-e97d-438a-93f9-85efa4d0e776","Type":"ContainerStarted","Data":"92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d"}
Sep 11 14:53:28.943144 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:28.943095    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" podStartSLOduration=1.2927809 podStartE2EDuration="3.943079269s" totalImagesPullingTime="2.650298369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:28.941948696 +0000 UTC m=+815.264577067" watchObservedRunningTime="2026-09-11 14:53:28.943079269 +0000 UTC m=+815.265707637"
Sep 11 14:53:29.002345 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:29.002306    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:29.575430 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:29.575392    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:29.579812 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:29.579786    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:29.923573 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:29.923488    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:29.924516 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:29.924499    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:29.989437 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:29.989402    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:30.311851 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:30.311809    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:30.410359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:30.410311    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:30.994821 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:30.994785    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"]
Sep 11 14:53:30.998583 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:30.998561    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.001831 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.001802    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzqqg\""
Sep 11 14:53:31.002150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.002116    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 14:53:31.003070 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.003045    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:53:31.003235 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.003124    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-3219e3-serving-cert\""
Sep 11 14:53:31.004893 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.004874    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 14:53:31.010780 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.010757    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"]
Sep 11 14:53:31.037749 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.037716    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:31.075101 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.075065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f07d25-e503-4fae-a9c0-63ac52053758-openshift-service-ca\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.075302 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.075106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f89g5\" (UniqueName: \"kubernetes.io/projected/f7f07d25-e503-4fae-a9c0-63ac52053758-kube-api-access-f89g5\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.075302 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.075136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-metrics-certs\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.075302 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.075235    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-maas-api-tls\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.138209 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.138169    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:31.176483 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.176425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f07d25-e503-4fae-a9c0-63ac52053758-openshift-service-ca\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.176659 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.176488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f89g5\" (UniqueName: \"kubernetes.io/projected/f7f07d25-e503-4fae-a9c0-63ac52053758-kube-api-access-f89g5\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.176659 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.176517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-metrics-certs\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.176659 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.176538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-maas-api-tls\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.176659 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:31.176629    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-serving-cert: secret "maas-api-e2e-autores-3219e3-serving-cert" not found
Sep 11 14:53:31.176897 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:31.176692    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-maas-api-tls podName:f7f07d25-e503-4fae-a9c0-63ac52053758 nodeName:}" failed. No retries permitted until 2026-09-11 14:53:31.676676162 +0000 UTC m=+817.999304510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-maas-api-tls") pod "maas-api-e2e-autores-3219e3-754df69985-w79cn" (UID: "f7f07d25-e503-4fae-a9c0-63ac52053758") : secret "maas-api-e2e-autores-3219e3-serving-cert" not found
Sep 11 14:53:31.177135 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.177114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f07d25-e503-4fae-a9c0-63ac52053758-openshift-service-ca\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.178873 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.178848    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-metrics-certs\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.185219 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.185191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f89g5\" (UniqueName: \"kubernetes.io/projected/f7f07d25-e503-4fae-a9c0-63ac52053758-kube-api-access-f89g5\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.466035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.465935    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:31.568533 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.568496    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:31.668750 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.668710    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:31.681473 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.681424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-maas-api-tls\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.683834 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.683811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-maas-api-tls\") pod \"maas-api-e2e-autores-3219e3-754df69985-w79cn\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:31.909405 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:31.909373    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:32.022317 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:32.022279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:32.182786 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:32.182758    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"]
Sep 11 14:53:32.184635 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:53:32.184609    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f07d25_e503_4fae_a9c0_63ac52053758.slice/crio-0494fc8bdad551030c902db43fadc12a204c0bd3de63cbb3b32c0c41257af302 WatchSource:0}: Error finding container 0494fc8bdad551030c902db43fadc12a204c0bd3de63cbb3b32c0c41257af302: Status 404 returned error can't find the container with id 0494fc8bdad551030c902db43fadc12a204c0bd3de63cbb3b32c0c41257af302
Sep 11 14:53:32.886732 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:32.886695    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:32.944759 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:32.944717    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn" event={"ID":"f7f07d25-e503-4fae-a9c0-63ac52053758","Type":"ContainerStarted","Data":"0494fc8bdad551030c902db43fadc12a204c0bd3de63cbb3b32c0c41257af302"}
Sep 11 14:53:33.107744 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.107687    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:33.206075 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.205992    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:33.541158 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.541121    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:33.636121 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.636084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:33.736161 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.736125    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"]
Sep 11 14:53:33.745005 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.744983    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:33.760997 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.750246    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f722bf59-serving-cert\""
Sep 11 14:53:33.760997 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.753558    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:33.783492 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.783305    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"]
Sep 11 14:53:33.800903 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.800819    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-maas-api-tls\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:33.801052 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.800897    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-metrics-certs\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:33.801052 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.800942    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-openshift-service-ca\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:33.801052 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.800981    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlmv\" (UniqueName: \"kubernetes.io/projected/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-kube-api-access-5hlmv\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:33.876897 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.876860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:33.902536 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.902501    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-maas-api-tls\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:33.902703 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.902541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-metrics-certs\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:33.902703 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.902575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-openshift-service-ca\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:33.902703 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.902606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlmv\" (UniqueName: \"kubernetes.io/projected/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-kube-api-access-5hlmv\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:33.902703 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:33.902674    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-serving-cert: secret "maas-api-e2e-ait-f722bf59-serving-cert" not found
Sep 11 14:53:33.902932 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:53:33.902749    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-maas-api-tls podName:ea5b57b2-8199-4b65-8391-9ca37a0ccc59 nodeName:}" failed. No retries permitted until 2026-09-11 14:53:34.402725869 +0000 UTC m=+820.725354221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-maas-api-tls") pod "maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l" (UID: "ea5b57b2-8199-4b65-8391-9ca37a0ccc59") : secret "maas-api-e2e-ait-f722bf59-serving-cert" not found
Sep 11 14:53:33.903302 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.903279    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-openshift-service-ca\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:33.905522 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.905442    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-metrics-certs\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:33.921604 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:33.921583    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlmv\" (UniqueName: \"kubernetes.io/projected/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-kube-api-access-5hlmv\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:34.005932 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.005899    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:34.157807 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.157715    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:34.336021 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.335985    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:53:34.408157 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.408071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-maas-api-tls\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:34.410762 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.410732    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-maas-api-tls\") pod \"maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:34.586529 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.586490    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:34.666211 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.666114    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:34.682420 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.682388    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:34.957706 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.957615    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn" event={"ID":"f7f07d25-e503-4fae-a9c0-63ac52053758","Type":"ContainerStarted","Data":"3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958"}
Sep 11 14:53:34.957706 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.957683    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:34.973389 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.973365    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"]
Sep 11 14:53:34.975226 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:53:34.975181    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5b57b2_8199_4b65_8391_9ca37a0ccc59.slice/crio-ebc7829b11181a7942a23222481dd348f481e9fc5229a20cc0ebb1ffc4186f0a WatchSource:0}: Error finding container ebc7829b11181a7942a23222481dd348f481e9fc5229a20cc0ebb1ffc4186f0a: Status 404 returned error can't find the container with id ebc7829b11181a7942a23222481dd348f481e9fc5229a20cc0ebb1ffc4186f0a
Sep 11 14:53:34.979236 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:34.979156    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn" podStartSLOduration=2.7351772309999998 podStartE2EDuration="4.979134234s" totalImagesPullingTime="2.243957003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:34.977190492 +0000 UTC m=+821.299818861" watchObservedRunningTime="2026-09-11 14:53:34.979134234 +0000 UTC m=+821.301762604"
Sep 11 14:53:35.134798 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:35.134752    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:35.964795 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:35.964753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l" event={"ID":"ea5b57b2-8199-4b65-8391-9ca37a0ccc59","Type":"ContainerStarted","Data":"1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17"}
Sep 11 14:53:35.965233 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:35.964814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l" event={"ID":"ea5b57b2-8199-4b65-8391-9ca37a0ccc59","Type":"ContainerStarted","Data":"ebc7829b11181a7942a23222481dd348f481e9fc5229a20cc0ebb1ffc4186f0a"}
Sep 11 14:53:35.965233 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:35.964856    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:35.989263 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:35.989218    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l" podStartSLOduration=2.71400937 podStartE2EDuration="2.989202105s" totalImagesPullingTime="275.192735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:35.987781765 +0000 UTC m=+822.310410152" watchObservedRunningTime="2026-09-11 14:53:35.989202105 +0000 UTC m=+822.311830474"
Sep 11 14:53:36.145632 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:36.145595    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:36.728190 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:36.728155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:37.129118 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:37.129037    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:37.947466 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:37.947425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:38.738127 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:38.738093    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:38.935861 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:38.935821    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:39.721176 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:39.721120    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:39.930249 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:39.930207    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:40.506396 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:40.506350    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:40.971008 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:40.970981    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:53:41.276917 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:41.276832    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:41.510204 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:41.510165    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:41.623265 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:41.623154    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:41.979535 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:41.979510    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:53:41.994138 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:41.994093    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:42.198712 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:42.198679    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:42.419756 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:42.419672    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:42.676721 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:42.676642    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:42.768625 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:42.768590    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:42.900695 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:42.900664    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:43.530154 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:43.530121    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:44.723057 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:44.723023    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:45.314199 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:45.314158    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:45.424079 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:45.424045    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:45.678526 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:45.677966    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:46.461617 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:46.461575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:47.315382 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:47.315344    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:47.550412 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:47.550371    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:48.499485 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:48.499433    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:49.490049 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.490019    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:49.495632 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.495588    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"]
Sep 11 14:53:49.502833 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.502808    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.505915 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.505615    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-90d6f5-serving-cert\""
Sep 11 14:53:49.513211 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.513187    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"]
Sep 11 14:53:49.583406 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.583372    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:49.648244 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.648206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwqq6\" (UniqueName: \"kubernetes.io/projected/a6129071-08b5-4e39-b0a9-340911efeaa7-kube-api-access-pwqq6\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.648477 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.648271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-metrics-certs\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.648477 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.648317    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6129071-08b5-4e39-b0a9-340911efeaa7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.648477 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.648403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-maas-api-tls\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.685022 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.684968    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:49.749682 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.749651    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-maas-api-tls\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.749854 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.749721    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwqq6\" (UniqueName: \"kubernetes.io/projected/a6129071-08b5-4e39-b0a9-340911efeaa7-kube-api-access-pwqq6\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.749854 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.749749    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-metrics-certs\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.749854 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.749780    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6129071-08b5-4e39-b0a9-340911efeaa7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.750418 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.750395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6129071-08b5-4e39-b0a9-340911efeaa7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.752195 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.752170    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-metrics-certs\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.752290 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.752170    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-maas-api-tls\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.763694 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.763670    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwqq6\" (UniqueName: \"kubernetes.io/projected/a6129071-08b5-4e39-b0a9-340911efeaa7-kube-api-access-pwqq6\") pod \"maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.806283 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.806249    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:49.816620 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.816590    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:49.908510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:49.908476    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:50.093679 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:50.093647    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"]
Sep 11 14:53:50.097079 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:53:50.097045    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6129071_08b5_4e39_b0a9_340911efeaa7.slice/crio-150d6644adb81532d1faba1b2d3f07537876e428f9a9d98c07e76260720937f7 WatchSource:0}: Error finding container 150d6644adb81532d1faba1b2d3f07537876e428f9a9d98c07e76260720937f7: Status 404 returned error can't find the container with id 150d6644adb81532d1faba1b2d3f07537876e428f9a9d98c07e76260720937f7
Sep 11 14:53:50.186466 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:50.186417    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:50.373885 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:50.373790    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:50.624401 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:50.624305    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:53:50.776098 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:50.776039    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:51.051121 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:51.051088    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc" event={"ID":"a6129071-08b5-4e39-b0a9-340911efeaa7","Type":"ContainerStarted","Data":"ef5c892707cee958f0a6d3a3a76bd2ad2182b5a3d13b166a32201572229c7682"}
Sep 11 14:53:51.051325 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:51.051161    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc" event={"ID":"a6129071-08b5-4e39-b0a9-340911efeaa7","Type":"ContainerStarted","Data":"150d6644adb81532d1faba1b2d3f07537876e428f9a9d98c07e76260720937f7"}
Sep 11 14:53:51.051325 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:51.051220    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:51.079035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:51.078974    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc" podStartSLOduration=1.768350648 podStartE2EDuration="2.078958684s" totalImagesPullingTime="310.608036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:51.076717699 +0000 UTC m=+837.399346068" watchObservedRunningTime="2026-09-11 14:53:51.078958684 +0000 UTC m=+837.401587052"
Sep 11 14:53:51.918151 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:51.918116    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:52.672420 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:52.672326    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:53.064247 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:53.064216    2559 generic.go:356] "Generic (PLEG): container finished" podID="6c163109-e97d-438a-93f9-85efa4d0e776" containerID="92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d" exitCode=0
Sep 11 14:53:53.064681 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:53.064298    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" event={"ID":"6c163109-e97d-438a-93f9-85efa4d0e776","Type":"ContainerDied","Data":"92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d"}
Sep 11 14:53:53.064901 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:53.064884    2559 scope.go:117] "RemoveContainer" containerID="92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d"
Sep 11 14:53:53.067681 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:53.067648    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:53:53.067793 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:53.067721    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:53:53.067793 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:53.067751    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:53:53.494573 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:53.494532    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:54.074801 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:54.074750    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" event={"ID":"6c163109-e97d-438a-93f9-85efa4d0e776","Type":"ContainerStarted","Data":"d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421"}
Sep 11 14:53:54.405034 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:54.404930    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:54.575264 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:54.575212    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:54.575905 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:54.575800    2559 patch_prober.go:28] interesting pod/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 14:53:54.576094 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:54.576067    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 14:53:55.217120 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:55.217089    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:55.580263 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:55.579829    2559 patch_prober.go:28] interesting pod/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 14:53:55.580263 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:55.579893    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 14:53:56.077672 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:56.077639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:56.576738 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:56.576258    2559 patch_prober.go:28] interesting pod/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 14:53:56.576738 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:56.576323    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 14:53:56.929565 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:56.929470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:57.074053 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:57.073749    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:53:57.350561 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:57.350519    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:57.451345 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:57.451311    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:57.575863 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:57.575818    2559 patch_prober.go:28] interesting pod/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 14:53:57.576067 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:57.575888    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 14:53:57.797342 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:57.797300    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:53:58.575717 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:58.575672    2559 patch_prober.go:28] interesting pod/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 14:53:58.575934 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:58.575743    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 14:53:59.579617 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:59.579589    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:59.579975 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:59.579856    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:53:59.580552 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:53:59.580535    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:54:01.352591 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.352559    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"]
Sep 11 14:54:01.353062 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.352810    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l" podUID="ea5b57b2-8199-4b65-8391-9ca37a0ccc59" containerName="maas-api" containerID="cri-o://1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17" gracePeriod=30
Sep 11 14:54:01.381387 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.381343    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:01.479894 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.479867    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:01.579304 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.579269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:01.597896 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.597868    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:54:01.663870 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.663799    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-metrics-certs\") pod \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") "
Sep 11 14:54:01.663870 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.663839    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-maas-api-tls\") pod \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") "
Sep 11 14:54:01.664074 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.663898    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-openshift-service-ca\") pod \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") "
Sep 11 14:54:01.664074 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.663953    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hlmv\" (UniqueName: \"kubernetes.io/projected/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-kube-api-access-5hlmv\") pod \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\" (UID: \"ea5b57b2-8199-4b65-8391-9ca37a0ccc59\") "
Sep 11 14:54:01.664284 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.664263    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea5b57b2-8199-4b65-8391-9ca37a0ccc59" (UID: "ea5b57b2-8199-4b65-8391-9ca37a0ccc59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:54:01.666100 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.666068    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-kube-api-access-5hlmv" (OuterVolumeSpecName: "kube-api-access-5hlmv") pod "ea5b57b2-8199-4b65-8391-9ca37a0ccc59" (UID: "ea5b57b2-8199-4b65-8391-9ca37a0ccc59"). InnerVolumeSpecName "kube-api-access-5hlmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:54:01.666211 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.666110    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea5b57b2-8199-4b65-8391-9ca37a0ccc59" (UID: "ea5b57b2-8199-4b65-8391-9ca37a0ccc59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:54:01.666211 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.666137    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea5b57b2-8199-4b65-8391-9ca37a0ccc59" (UID: "ea5b57b2-8199-4b65-8391-9ca37a0ccc59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:54:01.679167 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.679145    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:01.754871 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.754839    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"]
Sep 11 14:54:01.765402 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.765376    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hlmv\" (UniqueName: \"kubernetes.io/projected/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-kube-api-access-5hlmv\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:01.765402 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.765402    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:01.765402 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.765411    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:01.765701 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.765420    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5b57b2-8199-4b65-8391-9ca37a0ccc59-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:01.778717 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:01.778684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:02.005797 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.005767    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:02.119669 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.119633    2559 generic.go:356] "Generic (PLEG): container finished" podID="ea5b57b2-8199-4b65-8391-9ca37a0ccc59" containerID="1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17" exitCode=0
Sep 11 14:54:02.119828 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.119690    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"
Sep 11 14:54:02.119828 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.119714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l" event={"ID":"ea5b57b2-8199-4b65-8391-9ca37a0ccc59","Type":"ContainerDied","Data":"1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17"}
Sep 11 14:54:02.119828 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.119792    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l" event={"ID":"ea5b57b2-8199-4b65-8391-9ca37a0ccc59","Type":"ContainerDied","Data":"ebc7829b11181a7942a23222481dd348f481e9fc5229a20cc0ebb1ffc4186f0a"}
Sep 11 14:54:02.119828 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.119814    2559 scope.go:117] "RemoveContainer" containerID="1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17"
Sep 11 14:54:02.120176 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.120151    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy" containerID="cri-o://d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421" gracePeriod=30
Sep 11 14:54:02.128878 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.128865    2559 scope.go:117] "RemoveContainer" containerID="1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17"
Sep 11 14:54:02.129143 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:54:02.129126    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17\": container with ID starting with 1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17 not found: ID does not exist" containerID="1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17"
Sep 11 14:54:02.129192 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.129149    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17"} err="failed to get container status \"1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17\": rpc error: code = NotFound desc = could not find container \"1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17\": container with ID starting with 1a4c2f05ea566126aa592bde9b3395c321eb6bffaa7cd5cbbb5164b60ff52c17 not found: ID does not exist"
Sep 11 14:54:02.159151 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.159126    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"]
Sep 11 14:54:02.161928 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.161910    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f722bf59-c7869f88b-wqc5l"]
Sep 11 14:54:02.179767 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.179742    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea5b57b2-8199-4b65-8391-9ca37a0ccc59" path="/var/lib/kubelet/pods/ea5b57b2-8199-4b65-8391-9ca37a0ccc59/volumes"
Sep 11 14:54:02.294409 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.294307    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:02.418636 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.418605    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:02.574945 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.574855    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:02.830114 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.830040    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"]
Sep 11 14:54:02.830439 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.830425    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea5b57b2-8199-4b65-8391-9ca37a0ccc59" containerName="maas-api"
Sep 11 14:54:02.830439 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.830441    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea5b57b2-8199-4b65-8391-9ca37a0ccc59" containerName="maas-api"
Sep 11 14:54:02.830609 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.830515    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea5b57b2-8199-4b65-8391-9ca37a0ccc59" containerName="maas-api"
Sep 11 14:54:02.834106 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.834086    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.836959 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.836940    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4408ba1c-gw-openshift-default-dockercfg-qxghr\""
Sep 11 14:54:02.856396 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.856368    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"]
Sep 11 14:54:02.877188 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.877159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvwp\" (UniqueName: \"kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-kube-api-access-vfvwp\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.877308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.877201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-token\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.877308 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.877279    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-podinfo\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.877425 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.877316    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-certs\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.877425 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.877341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-envoy\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.877425 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.877367    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istiod-ca-cert\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.877603 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.877426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-data\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.877603 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.877502    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-credential-socket\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.877603 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.877536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-socket\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.886981 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.886955    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:02.978185 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978159    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-data\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978293 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-credential-socket\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978293 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978236    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-socket\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978405 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978301    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvwp\" (UniqueName: \"kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-kube-api-access-vfvwp\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978405 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-token\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978405 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978378    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-podinfo\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978587 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-certs\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978587 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-envoy\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978587 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istiod-ca-cert\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978734 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-credential-socket\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978791 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-socket\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978791 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978770    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-data\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.978900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.978806    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-certs\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.979201 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.979174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istiod-ca-cert\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.980606 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.980587    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-envoy\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.980963 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.980941    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-podinfo\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.988842 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.988797    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:02.991269 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.991245    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-token\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:02.991557 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:02.991541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvwp\" (UniqueName: \"kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-kube-api-access-vfvwp\") pod \"e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") " pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:03.086967 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:03.086883    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:03.145296 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:03.145269    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:03.420097 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:03.420072    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"]
Sep 11 14:54:03.421737 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:54:03.421710    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4e7c01_392a_4ea8_a24f_9100589e5b70.slice/crio-7cc3fd09bb62c206e3a3a79082bef9b48d2f138e2bcc2d015f5f8cb99ababc86 WatchSource:0}: Error finding container 7cc3fd09bb62c206e3a3a79082bef9b48d2f138e2bcc2d015f5f8cb99ababc86: Status 404 returned error can't find the container with id 7cc3fd09bb62c206e3a3a79082bef9b48d2f138e2bcc2d015f5f8cb99ababc86
Sep 11 14:54:03.424236 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:03.424201    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:54:03.424313 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:03.424272    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:54:03.424313 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:03.424304    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:54:03.902666 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:03.902636    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:04.132930 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:04.132889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" event={"ID":"ca4e7c01-392a-4ea8-a24f-9100589e5b70","Type":"ContainerStarted","Data":"cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e"}
Sep 11 14:54:04.133114 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:04.132949    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" event={"ID":"ca4e7c01-392a-4ea8-a24f-9100589e5b70","Type":"ContainerStarted","Data":"7cc3fd09bb62c206e3a3a79082bef9b48d2f138e2bcc2d015f5f8cb99ababc86"}
Sep 11 14:54:04.145948 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:04.145925    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:04.151329 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:04.151307    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:04.158524 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:04.158428    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" podStartSLOduration=2.158413778 podStartE2EDuration="2.158413778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:04.154434384 +0000 UTC m=+850.477062766" watchObservedRunningTime="2026-09-11 14:54:04.158413778 +0000 UTC m=+850.481042147"
Sep 11 14:54:04.605415 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:04.605377    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:05.140679 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:05.140644    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:05.141660 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:05.141640    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:05.306406 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:05.306349    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:05.636785 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:05.636749    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:05.737122 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:05.737085    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:05.820444 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:05.820411    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:54:06.143870 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:06.143833    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:06.886118 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:06.886087    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"]
Sep 11 14:54:06.893947 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:06.893923    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:06.897201 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:06.897183    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4408ba1c-serving-cert\""
Sep 11 14:54:06.904106 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:06.904082    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"]
Sep 11 14:54:06.912769 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:06.912747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-maas-api-tls\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:06.912897 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:06.912781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64dabd0-8fd0-4deb-a6b0-70487971c397-openshift-service-ca\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:06.912965 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:06.912945    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-metrics-certs\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:06.913017 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:06.912997    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g58j2\" (UniqueName: \"kubernetes.io/projected/a64dabd0-8fd0-4deb-a6b0-70487971c397-kube-api-access-g58j2\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:06.915894 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:06.915867    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:07.013995 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.013951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-metrics-certs\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:07.014189 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.014013    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g58j2\" (UniqueName: \"kubernetes.io/projected/a64dabd0-8fd0-4deb-a6b0-70487971c397-kube-api-access-g58j2\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:07.014189 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.014062    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-maas-api-tls\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:07.014189 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.014084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64dabd0-8fd0-4deb-a6b0-70487971c397-openshift-service-ca\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:07.014347 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:54:07.014188    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-serving-cert: secret "maas-api-e2e-migrate-4408ba1c-serving-cert" not found
Sep 11 14:54:07.014347 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:54:07.014263    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-maas-api-tls podName:a64dabd0-8fd0-4deb-a6b0-70487971c397 nodeName:}" failed. No retries permitted until 2026-09-11 14:54:07.514246167 +0000 UTC m=+853.836874529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-maas-api-tls") pod "maas-api-e2e-migrate-4408ba1c-754795b797-wrldb" (UID: "a64dabd0-8fd0-4deb-a6b0-70487971c397") : secret "maas-api-e2e-migrate-4408ba1c-serving-cert" not found
Sep 11 14:54:07.014840 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.014811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64dabd0-8fd0-4deb-a6b0-70487971c397-openshift-service-ca\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:07.014985 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.014895    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:07.016606 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.016586    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-metrics-certs\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:07.027363 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.027342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g58j2\" (UniqueName: \"kubernetes.io/projected/a64dabd0-8fd0-4deb-a6b0-70487971c397-kube-api-access-g58j2\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:07.149889 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:54:07.149863    2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c163109_e97d_438a_93f9_85efa4d0e776.slice/crio-conmon-d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 14:54:07.200167 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.200135    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:07.371686 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.371657    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:54:07.416193 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416123    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-socket\") pod \"6c163109-e97d-438a-93f9-85efa4d0e776\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") "
Sep 11 14:54:07.416193 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416158    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c163109-e97d-438a-93f9-85efa4d0e776-istiod-ca-cert\") pod \"6c163109-e97d-438a-93f9-85efa4d0e776\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") "
Sep 11 14:54:07.416377 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416225    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-data\") pod \"6c163109-e97d-438a-93f9-85efa4d0e776\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") "
Sep 11 14:54:07.416377 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416255    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c163109-e97d-438a-93f9-85efa4d0e776-istio-podinfo\") pod \"6c163109-e97d-438a-93f9-85efa4d0e776\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") "
Sep 11 14:54:07.416377 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416272    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-certs\") pod \"6c163109-e97d-438a-93f9-85efa4d0e776\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") "
Sep 11 14:54:07.416377 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416311    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-envoy\") pod \"6c163109-e97d-438a-93f9-85efa4d0e776\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") "
Sep 11 14:54:07.416606 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416386    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c163109-e97d-438a-93f9-85efa4d0e776" (UID: "6c163109-e97d-438a-93f9-85efa4d0e776"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:54:07.416606 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416434    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgxfb\" (UniqueName: \"kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-kube-api-access-bgxfb\") pod \"6c163109-e97d-438a-93f9-85efa4d0e776\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") "
Sep 11 14:54:07.416606 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416504    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-credential-socket\") pod \"6c163109-e97d-438a-93f9-85efa4d0e776\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") "
Sep 11 14:54:07.416606 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416537    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-istio-token\") pod \"6c163109-e97d-438a-93f9-85efa4d0e776\" (UID: \"6c163109-e97d-438a-93f9-85efa4d0e776\") "
Sep 11 14:54:07.416810 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416627    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c163109-e97d-438a-93f9-85efa4d0e776" (UID: "6c163109-e97d-438a-93f9-85efa4d0e776"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:54:07.416810 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416636    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c163109-e97d-438a-93f9-85efa4d0e776" (UID: "6c163109-e97d-438a-93f9-85efa4d0e776"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:54:07.416810 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416738    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c163109-e97d-438a-93f9-85efa4d0e776-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c163109-e97d-438a-93f9-85efa4d0e776" (UID: "6c163109-e97d-438a-93f9-85efa4d0e776"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:54:07.416978 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416823    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c163109-e97d-438a-93f9-85efa4d0e776" (UID: "6c163109-e97d-438a-93f9-85efa4d0e776"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:54:07.416978 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416970    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-credential-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:07.417087 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.416991    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:07.417087 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.417006    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c163109-e97d-438a-93f9-85efa4d0e776-istiod-ca-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:07.417087 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.417020    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-data\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:07.417087 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.417031    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-workload-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:07.419117 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.419093    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c163109-e97d-438a-93f9-85efa4d0e776" (UID: "6c163109-e97d-438a-93f9-85efa4d0e776"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:54:07.419117 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.419105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c163109-e97d-438a-93f9-85efa4d0e776-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c163109-e97d-438a-93f9-85efa4d0e776" (UID: "6c163109-e97d-438a-93f9-85efa4d0e776"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:54:07.419250 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.419161    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c163109-e97d-438a-93f9-85efa4d0e776" (UID: "6c163109-e97d-438a-93f9-85efa4d0e776"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:54:07.419250 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.419180    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-kube-api-access-bgxfb" (OuterVolumeSpecName: "kube-api-access-bgxfb") pod "6c163109-e97d-438a-93f9-85efa4d0e776" (UID: "6c163109-e97d-438a-93f9-85efa4d0e776"). InnerVolumeSpecName "kube-api-access-bgxfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:54:07.518494 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.518465    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-maas-api-tls\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:07.518635 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.518570    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c163109-e97d-438a-93f9-85efa4d0e776-istio-podinfo\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:07.518635 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.518580    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c163109-e97d-438a-93f9-85efa4d0e776-istio-envoy\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:07.518635 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.518589    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgxfb\" (UniqueName: \"kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-kube-api-access-bgxfb\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:07.518635 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.518598    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c163109-e97d-438a-93f9-85efa4d0e776-istio-token\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:07.520678 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.520648    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-maas-api-tls\") pod \"maas-api-e2e-migrate-4408ba1c-754795b797-wrldb\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:07.564970 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.564937    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:07.671399 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.671316    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:07.806563 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:07.806531    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:08.009903 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.009872    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:08.082002 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.081978    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"]
Sep 11 14:54:08.083706 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:54:08.083678    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64dabd0_8fd0_4deb_a6b0_70487971c397.slice/crio-c40cc645b14b49d14538a1037e5017b161046be4383db7cd57c23c503c694a31 WatchSource:0}: Error finding container c40cc645b14b49d14538a1037e5017b161046be4383db7cd57c23c503c694a31: Status 404 returned error can't find the container with id c40cc645b14b49d14538a1037e5017b161046be4383db7cd57c23c503c694a31
Sep 11 14:54:08.156936 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.156907    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb" event={"ID":"a64dabd0-8fd0-4deb-a6b0-70487971c397","Type":"ContainerStarted","Data":"c40cc645b14b49d14538a1037e5017b161046be4383db7cd57c23c503c694a31"}
Sep 11 14:54:08.158835 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.158812    2559 generic.go:356] "Generic (PLEG): container finished" podID="6c163109-e97d-438a-93f9-85efa4d0e776" containerID="d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421" exitCode=0
Sep 11 14:54:08.158918 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.158849    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" event={"ID":"6c163109-e97d-438a-93f9-85efa4d0e776","Type":"ContainerDied","Data":"d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421"}
Sep 11 14:54:08.158918 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.158876    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"
Sep 11 14:54:08.158918 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.158889    2559 scope.go:117] "RemoveContainer" containerID="d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421"
Sep 11 14:54:08.159063 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.158880    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd" event={"ID":"6c163109-e97d-438a-93f9-85efa4d0e776","Type":"ContainerDied","Data":"b0b1ac2764bcedbe6ee302d8dba0ddc5643b28226d90ea21979256e96b1d1f6d"}
Sep 11 14:54:08.168297 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.168266    2559 scope.go:117] "RemoveContainer" containerID="92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d"
Sep 11 14:54:08.176315 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.176297    2559 scope.go:117] "RemoveContainer" containerID="d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421"
Sep 11 14:54:08.176627 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:54:08.176610    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421\": container with ID starting with d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421 not found: ID does not exist" containerID="d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421"
Sep 11 14:54:08.176698 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.176635    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421"} err="failed to get container status \"d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421\": rpc error: code = NotFound desc = could not find container \"d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421\": container with ID starting with d93bb72458879c80b013ff2b20c2e6cbb2c347087b58e7247934d29a6d945421 not found: ID does not exist"
Sep 11 14:54:08.176698 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.176653    2559 scope.go:117] "RemoveContainer" containerID="92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d"
Sep 11 14:54:08.176880 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:54:08.176861    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d\": container with ID starting with 92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d not found: ID does not exist" containerID="92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d"
Sep 11 14:54:08.176922 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.176886    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d"} err="failed to get container status \"92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d\": rpc error: code = NotFound desc = could not find container \"92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d\": container with ID starting with 92c1078a5a8e3aa8f292b561d6fd01624e9ee29856b209036b34b9a1cdb43d8d not found: ID does not exist"
Sep 11 14:54:08.185650 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.185630    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"]
Sep 11 14:54:08.189100 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.189081    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f722bf59-openshift-default-d8b977858-fcsxd"]
Sep 11 14:54:08.902084 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:08.902052    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:09.167911 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:09.167823    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb" event={"ID":"a64dabd0-8fd0-4deb-a6b0-70487971c397","Type":"ContainerStarted","Data":"51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b"}
Sep 11 14:54:09.168294 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:09.167952    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:09.188750 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:09.188698    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb" podStartSLOduration=2.851927486 podStartE2EDuration="3.188676953s" totalImagesPullingTime="336.749467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:09.185964959 +0000 UTC m=+855.508593331" watchObservedRunningTime="2026-09-11 14:54:09.188676953 +0000 UTC m=+855.511305323"
Sep 11 14:54:09.636409 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:09.636369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:10.179495 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:10.179439    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" path="/var/lib/kubelet/pods/6c163109-e97d-438a-93f9-85efa4d0e776/volumes"
Sep 11 14:54:10.287656 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:10.287606    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:10.383218 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:10.383156    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:11.071293 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:11.071258    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:11.799847 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:11.799811    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:12.609833 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:12.609790    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:13.498082 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:13.498041    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:13.596388 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:13.596348    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:15.178926 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:15.178894    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:16.073439 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.073391    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"]
Sep 11 14:54:16.074076 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.074051    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy"
Sep 11 14:54:16.074076 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.074077    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy"
Sep 11 14:54:16.074224 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.074112    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy"
Sep 11 14:54:16.074224 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.074122    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy"
Sep 11 14:54:16.074224 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.074211    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy"
Sep 11 14:54:16.074224 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.074220    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c163109-e97d-438a-93f9-85efa4d0e776" containerName="istio-proxy"
Sep 11 14:54:16.079085 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.079061    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.083297 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.083256    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"kube-root-ca.crt\""
Sep 11 14:54:16.083631 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.083609    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"openshift-service-ca.crt\""
Sep 11 14:54:16.083848 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.083822    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"default-dockercfg-rhxhm\""
Sep 11 14:54:16.084222 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.084185    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"simulator-w0-e7ca56-kserve-self-signed-certs\""
Sep 11 14:54:16.094219 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.094190    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"]
Sep 11 14:54:16.098942 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.098916    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tmp-dir\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.099685 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.099443    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-model-cache\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.099685 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.099513    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-home\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.099685 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.099567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6qwt\" (UniqueName: \"kubernetes.io/projected/1f02f1ac-5778-4554-bfe0-8d03a17480dc-kube-api-access-c6qwt\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.099685 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.099607    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-dshm\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.099685 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.099678    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tls-certs\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.187919 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.187862    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:16.201117 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.201088    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-model-cache\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.201117 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.201118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-home\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.201326 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.201162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6qwt\" (UniqueName: \"kubernetes.io/projected/1f02f1ac-5778-4554-bfe0-8d03a17480dc-kube-api-access-c6qwt\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.201326 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.201206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-dshm\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.201326 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.201301    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tls-certs\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.201556 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.201524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-home\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.201817 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.201793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tmp-dir\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.201979 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.201953    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-model-cache\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.202116 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.202089    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tmp-dir\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.203640 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.203620    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-dshm\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.203985 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.203965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tls-certs\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.210674 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.210649    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6qwt\" (UniqueName: \"kubernetes.io/projected/1f02f1ac-5778-4554-bfe0-8d03a17480dc-kube-api-access-c6qwt\") pod \"simulator-w0-e7ca56-kserve-b76fdf748-dmcbr\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") " pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.400897 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.400816    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:16.681703 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:16.681669    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"]
Sep 11 14:54:17.213948 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:17.213913    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr" event={"ID":"1f02f1ac-5778-4554-bfe0-8d03a17480dc","Type":"ContainerStarted","Data":"abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846"}
Sep 11 14:54:17.214316 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:17.213958    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr" event={"ID":"1f02f1ac-5778-4554-bfe0-8d03a17480dc","Type":"ContainerStarted","Data":"e43c00cae392a263db8eb767726366bc6dad57e173dfba72731f86a3f2c662c9"}
Sep 11 14:54:17.235198 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:17.235141    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr" podStartSLOduration=1.235126259 podStartE2EDuration="1.235126259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:17.233998545 +0000 UTC m=+863.556626914" watchObservedRunningTime="2026-09-11 14:54:17.235126259 +0000 UTC m=+863.557754627"
Sep 11 14:54:21.430897 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:21.430860    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:54:24.255519 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:24.255488    2559 generic.go:356] "Generic (PLEG): container finished" podID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerID="cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e" exitCode=0
Sep 11 14:54:24.255933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:24.255567    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" event={"ID":"ca4e7c01-392a-4ea8-a24f-9100589e5b70","Type":"ContainerDied","Data":"cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e"}
Sep 11 14:54:24.256054 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:24.256038    2559 scope.go:117] "RemoveContainer" containerID="cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e"
Sep 11 14:54:24.257204 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:24.257176    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:54:24.257273 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:24.257246    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:54:24.257317 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:24.257275    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:54:25.266166 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:25.266124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" event={"ID":"ca4e7c01-392a-4ea8-a24f-9100589e5b70","Type":"ContainerStarted","Data":"05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991"}
Sep 11 14:54:26.146331 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:26.146292    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:26.146709 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:26.146685    2559 patch_prober.go:28] interesting pod/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 11 14:54:26.146760 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:26.146735    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 11 14:54:26.401204 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:26.401120    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:26.401204 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:26.401159    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:26.413648 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:26.413615    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:27.146650 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:27.146612    2559 patch_prober.go:28] interesting pod/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 11 14:54:27.146850 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:27.146708    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 11 14:54:27.294471 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:27.294422    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 14:54:28.146263 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:28.146225    2559 patch_prober.go:28] interesting pod/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 11 14:54:28.146682 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:28.146285    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 11 14:54:29.104913 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:29.104881    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:29.145899 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:29.145872    2559 patch_prober.go:28] interesting pod/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 11 14:54:29.146031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:29.145917    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 11 14:54:29.999884 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:29.999850    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:30.146171 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:30.146137    2559 patch_prober.go:28] interesting pod/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 11 14:54:30.146339 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:30.146188    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 11 14:54:30.705429 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:30.705399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:31.150024 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:31.149989    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:31.150493 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:31.150233    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:31.151080 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:31.151059    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:31.482145 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:31.482108    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:32.319345 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:32.319302    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:33.019159 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:33.019129    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:33.851877 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:33.851845    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:34.752333 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:34.752302    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:35.509370 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:35.509335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:36.301088 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:36.301043    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:36.886558 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:36.886529    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"]
Sep 11 14:54:36.886954 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:36.886743    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb" podUID="a64dabd0-8fd0-4deb-a6b0-70487971c397" containerName="maas-api" containerID="cri-o://51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b" gracePeriod=30
Sep 11 14:54:36.913302 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:36.913270    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:37.008917 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.008891    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:37.108796 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.108763    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:37.129259 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.129237    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:37.205270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.205194    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64dabd0-8fd0-4deb-a6b0-70487971c397-openshift-service-ca\") pod \"a64dabd0-8fd0-4deb-a6b0-70487971c397\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") "
Sep 11 14:54:37.205270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.205256    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-metrics-certs\") pod \"a64dabd0-8fd0-4deb-a6b0-70487971c397\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") "
Sep 11 14:54:37.205525 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.205290    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-maas-api-tls\") pod \"a64dabd0-8fd0-4deb-a6b0-70487971c397\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") "
Sep 11 14:54:37.205525 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.205440    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g58j2\" (UniqueName: \"kubernetes.io/projected/a64dabd0-8fd0-4deb-a6b0-70487971c397-kube-api-access-g58j2\") pod \"a64dabd0-8fd0-4deb-a6b0-70487971c397\" (UID: \"a64dabd0-8fd0-4deb-a6b0-70487971c397\") "
Sep 11 14:54:37.205640 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.205621    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64dabd0-8fd0-4deb-a6b0-70487971c397-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a64dabd0-8fd0-4deb-a6b0-70487971c397" (UID: "a64dabd0-8fd0-4deb-a6b0-70487971c397"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:54:37.205827 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.205809    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64dabd0-8fd0-4deb-a6b0-70487971c397-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:37.207293 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.207270    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a64dabd0-8fd0-4deb-a6b0-70487971c397" (UID: "a64dabd0-8fd0-4deb-a6b0-70487971c397"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:54:37.207374 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.207309    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a64dabd0-8fd0-4deb-a6b0-70487971c397" (UID: "a64dabd0-8fd0-4deb-a6b0-70487971c397"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:54:37.207374 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.207343    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64dabd0-8fd0-4deb-a6b0-70487971c397-kube-api-access-g58j2" (OuterVolumeSpecName: "kube-api-access-g58j2") pod "a64dabd0-8fd0-4deb-a6b0-70487971c397" (UID: "a64dabd0-8fd0-4deb-a6b0-70487971c397"). InnerVolumeSpecName "kube-api-access-g58j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:54:37.211180 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.211152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:37.283468 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.283418    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"]
Sep 11 14:54:37.283990 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.283935    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy" containerID="cri-o://05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991" gracePeriod=30
Sep 11 14:54:37.288420 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.288386    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:37.307378 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.307356    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:37.307378 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.307376    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g58j2\" (UniqueName: \"kubernetes.io/projected/a64dabd0-8fd0-4deb-a6b0-70487971c397-kube-api-access-g58j2\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:37.307613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.307385    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64dabd0-8fd0-4deb-a6b0-70487971c397-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:37.333188 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.333161    2559 generic.go:356] "Generic (PLEG): container finished" podID="a64dabd0-8fd0-4deb-a6b0-70487971c397" containerID="51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b" exitCode=0
Sep 11 14:54:37.333302 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.333211    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"
Sep 11 14:54:37.333302 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.333225    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb" event={"ID":"a64dabd0-8fd0-4deb-a6b0-70487971c397","Type":"ContainerDied","Data":"51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b"}
Sep 11 14:54:37.333302 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.333264    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb" event={"ID":"a64dabd0-8fd0-4deb-a6b0-70487971c397","Type":"ContainerDied","Data":"c40cc645b14b49d14538a1037e5017b161046be4383db7cd57c23c503c694a31"}
Sep 11 14:54:37.333302 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.333284    2559 scope.go:117] "RemoveContainer" containerID="51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b"
Sep 11 14:54:37.343556 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.343432    2559 scope.go:117] "RemoveContainer" containerID="51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b"
Sep 11 14:54:37.343746 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:54:37.343725    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b\": container with ID starting with 51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b not found: ID does not exist" containerID="51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b"
Sep 11 14:54:37.343815 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.343758    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b"} err="failed to get container status \"51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b\": rpc error: code = NotFound desc = could not find container \"51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b\": container with ID starting with 51496b5645b322bd0319b39b86d9b5fc98557d7fbc31ca1c76bc0b3e93f42d9b not found: ID does not exist"
Sep 11 14:54:37.358259 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.358238    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"]
Sep 11 14:54:37.362548 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.362527    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4408ba1c-754795b797-wrldb"]
Sep 11 14:54:37.581935 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.581905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:37.779251 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:37.779205    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:38.093688 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.093655    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"]
Sep 11 14:54:38.094146 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.094098    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a64dabd0-8fd0-4deb-a6b0-70487971c397" containerName="maas-api"
Sep 11 14:54:38.094146 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.094112    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a64dabd0-8fd0-4deb-a6b0-70487971c397" containerName="maas-api"
Sep 11 14:54:38.094257 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.094178    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a64dabd0-8fd0-4deb-a6b0-70487971c397" containerName="maas-api"
Sep 11 14:54:38.096774 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.096754    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.099634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.099613    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c0b7dbbd-openshift-default-dockercfg-fsc46\""
Sep 11 14:54:38.111813 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.111788    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"]
Sep 11 14:54:38.122475 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.122434    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:38.180599 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.180571    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a64dabd0-8fd0-4deb-a6b0-70487971c397" path="/var/lib/kubelet/pods/a64dabd0-8fd0-4deb-a6b0-70487971c397/volumes"
Sep 11 14:54:38.216910 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.216884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdtd\" (UniqueName: \"kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-kube-api-access-pqdtd\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.217052 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.216914    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-credential-socket\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.217052 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.216954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24c5901-9e3b-43b6-abd5-52f769464edc-istiod-ca-cert\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.217052 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.217027    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-socket\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.217164 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.217062    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-data\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.217164 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.217091    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-podinfo\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.217230 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.217168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-envoy\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.217230 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.217196    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-certs\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.217230 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.217222    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-token\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.294552 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.294523    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:38.318323 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.318295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-envoy\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.318478 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.318330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-certs\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.318478 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.318350    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-token\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.318478 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.318389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdtd\" (UniqueName: \"kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-kube-api-access-pqdtd\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.318478 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.318415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-credential-socket\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.318680 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.318505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24c5901-9e3b-43b6-abd5-52f769464edc-istiod-ca-cert\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.318680 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.318564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-socket\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.318680 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.318591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-data\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.318680 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.318622    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-podinfo\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.318973 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.318949    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-credential-socket\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.319055 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.318980    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-socket\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.319055 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.319018    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-data\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.319163 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.319086    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-certs\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.319427 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.319407    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24c5901-9e3b-43b6-abd5-52f769464edc-istiod-ca-cert\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.320551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.320533    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-envoy\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.320916 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.320901    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-podinfo\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.327186 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.327160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-token\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.327289 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.327274    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdtd\" (UniqueName: \"kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-kube-api-access-pqdtd\") pod \"e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") " pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.406741 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.406654    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:38.458186 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.458118    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:38.678858 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.678829    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"]
Sep 11 14:54:38.680810 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:54:38.680785    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24c5901_9e3b_43b6_abd5_52f769464edc.slice/crio-d9e7bca2a64f4405240677086089992485cf3ee88e681182b0a184209e1df8f8 WatchSource:0}: Error finding container d9e7bca2a64f4405240677086089992485cf3ee88e681182b0a184209e1df8f8: Status 404 returned error can't find the container with id d9e7bca2a64f4405240677086089992485cf3ee88e681182b0a184209e1df8f8
Sep 11 14:54:38.683076 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.683045    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:54:38.683153 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.683108    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:54:38.683153 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:38.683135    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:54:39.152505 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:39.152447    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:39.346208 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:39.346169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g" event={"ID":"e24c5901-9e3b-43b6-abd5-52f769464edc","Type":"ContainerStarted","Data":"1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb"}
Sep 11 14:54:39.346367 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:39.346238    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g" event={"ID":"e24c5901-9e3b-43b6-abd5-52f769464edc","Type":"ContainerStarted","Data":"d9e7bca2a64f4405240677086089992485cf3ee88e681182b0a184209e1df8f8"}
Sep 11 14:54:39.370011 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:39.369963    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g" podStartSLOduration=1.36994559 podStartE2EDuration="1.36994559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:39.367717078 +0000 UTC m=+885.690345447" watchObservedRunningTime="2026-09-11 14:54:39.36994559 +0000 UTC m=+885.692573959"
Sep 11 14:54:39.407034 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:39.406953    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:39.411960 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:39.411937    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:40.019885 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:40.019846    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:40.351002 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:40.350913    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:40.351789 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:40.351771    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:54:40.724322 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:40.724282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:40.824277 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:40.824233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:41.112762 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.112690    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:41.465029 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.464942    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"]
Sep 11 14:54:41.467366 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.467341    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.470357 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.470334    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"default-dockercfg-cz84q\""
Sep 11 14:54:41.471883 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.471821    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"kube-root-ca.crt\""
Sep 11 14:54:41.471883 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.471836    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"simulator-w4-837632-kserve-self-signed-certs\""
Sep 11 14:54:41.472061 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.471916    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"openshift-service-ca.crt\""
Sep 11 14:54:41.481630 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.481605    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"]
Sep 11 14:54:41.499771 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.499735    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:41.550016 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.549990    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tmp-dir\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.550127 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.550020    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tls-certs\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.550127 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.550042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-model-cache\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.550127 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.550113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-dshm\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.550244 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.550160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-home\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.550325 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.550245    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmsl4\" (UniqueName: \"kubernetes.io/projected/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-kube-api-access-vmsl4\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.651622 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.651591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmsl4\" (UniqueName: \"kubernetes.io/projected/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-kube-api-access-vmsl4\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.651761 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.651645    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tmp-dir\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.651761 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.651742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tls-certs\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.651839 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.651761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-model-cache\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.651839 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.651785    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-dshm\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.651839 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.651800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-home\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.652150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.652124    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tmp-dir\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.652150 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.652141    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-model-cache\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.652271 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.652189    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-home\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.654056 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.654032    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-dshm\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.654258 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.654240    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tls-certs\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.660305 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.660283    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmsl4\" (UniqueName: \"kubernetes.io/projected/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-kube-api-access-vmsl4\") pod \"simulator-w4-837632-kserve-74594cf674-67mmk\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") " pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.802344 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.802307    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:41.969954 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:41.969917    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:42.077898 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.077871    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"]
Sep 11 14:54:42.079911 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:54:42.079884    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f2bdc5c_96ce_4b48_ae4a_66b96dc4259c.slice/crio-e98c3541999297d04784ab74f7aec99abdda48e10d9376111b0fa6329e6008d8 WatchSource:0}: Error finding container e98c3541999297d04784ab74f7aec99abdda48e10d9376111b0fa6329e6008d8: Status 404 returned error can't find the container with id e98c3541999297d04784ab74f7aec99abdda48e10d9376111b0fa6329e6008d8
Sep 11 14:54:42.363179 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.363097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk" event={"ID":"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c","Type":"ContainerStarted","Data":"5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1"}
Sep 11 14:54:42.363179 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.363146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk" event={"ID":"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c","Type":"ContainerStarted","Data":"e98c3541999297d04784ab74f7aec99abdda48e10d9376111b0fa6329e6008d8"}
Sep 11 14:54:42.381922 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.381879    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk" podStartSLOduration=1.38186399 podStartE2EDuration="1.38186399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:42.380258397 +0000 UTC m=+888.702886767" watchObservedRunningTime="2026-09-11 14:54:42.38186399 +0000 UTC m=+888.704492359"
Sep 11 14:54:42.523639 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.523616    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:42.663571 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.663484    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-socket\") pod \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") "
Sep 11 14:54:42.663571 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.663539    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istiod-ca-cert\") pod \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") "
Sep 11 14:54:42.663571 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.663562    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-certs\") pod \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") "
Sep 11 14:54:42.663832 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.663623    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-token\") pod \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") "
Sep 11 14:54:42.663832 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.663650    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-envoy\") pod \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") "
Sep 11 14:54:42.663832 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.663683    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfvwp\" (UniqueName: \"kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-kube-api-access-vfvwp\") pod \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") "
Sep 11 14:54:42.663832 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.663705    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-podinfo\") pod \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") "
Sep 11 14:54:42.663832 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.663759    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-credential-socket\") pod \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") "
Sep 11 14:54:42.663832 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.663829    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-data\") pod \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\" (UID: \"ca4e7c01-392a-4ea8-a24f-9100589e5b70\") "
Sep 11 14:54:42.664108 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.663827    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca4e7c01-392a-4ea8-a24f-9100589e5b70" (UID: "ca4e7c01-392a-4ea8-a24f-9100589e5b70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:54:42.664163 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.664113    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca4e7c01-392a-4ea8-a24f-9100589e5b70" (UID: "ca4e7c01-392a-4ea8-a24f-9100589e5b70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:54:42.664217 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.664174    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:42.664816 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.664638    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca4e7c01-392a-4ea8-a24f-9100589e5b70" (UID: "ca4e7c01-392a-4ea8-a24f-9100589e5b70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:54:42.664816 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.664737    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca4e7c01-392a-4ea8-a24f-9100589e5b70" (UID: "ca4e7c01-392a-4ea8-a24f-9100589e5b70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:54:42.664816 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.664780    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca4e7c01-392a-4ea8-a24f-9100589e5b70" (UID: "ca4e7c01-392a-4ea8-a24f-9100589e5b70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:54:42.666733 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.666701    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca4e7c01-392a-4ea8-a24f-9100589e5b70" (UID: "ca4e7c01-392a-4ea8-a24f-9100589e5b70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:54:42.667367 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.667335    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-kube-api-access-vfvwp" (OuterVolumeSpecName: "kube-api-access-vfvwp") pod "ca4e7c01-392a-4ea8-a24f-9100589e5b70" (UID: "ca4e7c01-392a-4ea8-a24f-9100589e5b70"). InnerVolumeSpecName "kube-api-access-vfvwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:54:42.667689 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.667664    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca4e7c01-392a-4ea8-a24f-9100589e5b70" (UID: "ca4e7c01-392a-4ea8-a24f-9100589e5b70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:54:42.668400 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.668139    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca4e7c01-392a-4ea8-a24f-9100589e5b70" (UID: "ca4e7c01-392a-4ea8-a24f-9100589e5b70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:54:42.677139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.677115    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"]
Sep 11 14:54:42.677598 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.677579    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy"
Sep 11 14:54:42.677598 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.677599    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy"
Sep 11 14:54:42.677778 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.677695    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy"
Sep 11 14:54:42.679739 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.679723    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.682199 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.682183    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c0b7dbbd-serving-cert\""
Sep 11 14:54:42.693332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.693306    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"]
Sep 11 14:54:42.713943 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.713907    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:42.765086 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-maas-api-tls\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.765221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765127    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-metrics-certs\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.765221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765154    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w727l\" (UniqueName: \"kubernetes.io/projected/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-kube-api-access-w727l\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.765296 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765235    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-openshift-service-ca\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.765330 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765306    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-workload-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:42.765330 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765316    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istiod-ca-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:42.765330 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765324    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-token\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:42.765426 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765331    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-envoy\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:42.765426 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765341    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvwp\" (UniqueName: \"kubernetes.io/projected/ca4e7c01-392a-4ea8-a24f-9100589e5b70-kube-api-access-vfvwp\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:42.765426 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765363    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-podinfo\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:42.765426 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765373    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-credential-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:42.765426 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.765381    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4e7c01-392a-4ea8-a24f-9100589e5b70-istio-data\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:54:42.813898 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.813867    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:42.866145 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.866112    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-metrics-certs\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.866300 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.866168    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w727l\" (UniqueName: \"kubernetes.io/projected/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-kube-api-access-w727l\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.866300 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.866256    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-openshift-service-ca\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.866407 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.866304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-maas-api-tls\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.866484 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:54:42.866432    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-serving-cert: secret "maas-api-e2e-ait-c0b7dbbd-serving-cert" not found
Sep 11 14:54:42.866537 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:54:42.866517    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-maas-api-tls podName:9458c3e6-e1cb-4ae3-9eea-b3afef107e8e nodeName:}" failed. No retries permitted until 2026-09-11 14:54:43.366496875 +0000 UTC m=+889.689125242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-maas-api-tls") pod "maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6" (UID: "9458c3e6-e1cb-4ae3-9eea-b3afef107e8e") : secret "maas-api-e2e-ait-c0b7dbbd-serving-cert" not found
Sep 11 14:54:42.866904 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.866877    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-openshift-service-ca\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.868578 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.868561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-metrics-certs\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.878248 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.878217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w727l\" (UniqueName: \"kubernetes.io/projected/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-kube-api-access-w727l\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:42.925627 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:42.925555    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:43.136991 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.136954    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:54:43.321913 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.321880    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:43.371551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.371513    2559 generic.go:356] "Generic (PLEG): container finished" podID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerID="05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991" exitCode=0
Sep 11 14:54:43.371551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.371550    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"
Sep 11 14:54:43.371782 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.371597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" event={"ID":"ca4e7c01-392a-4ea8-a24f-9100589e5b70","Type":"ContainerDied","Data":"05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991"}
Sep 11 14:54:43.371782 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.371648    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9" event={"ID":"ca4e7c01-392a-4ea8-a24f-9100589e5b70","Type":"ContainerDied","Data":"7cc3fd09bb62c206e3a3a79082bef9b48d2f138e2bcc2d015f5f8cb99ababc86"}
Sep 11 14:54:43.371782 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.371676    2559 scope.go:117] "RemoveContainer" containerID="05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991"
Sep 11 14:54:43.372126 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.372106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-maas-api-tls\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:43.374609 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.374587    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-maas-api-tls\") pod \"maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:43.380297 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.380282    2559 scope.go:117] "RemoveContainer" containerID="cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e"
Sep 11 14:54:43.387841 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.387822    2559 scope.go:117] "RemoveContainer" containerID="05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991"
Sep 11 14:54:43.388106 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:54:43.388089    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991\": container with ID starting with 05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991 not found: ID does not exist" containerID="05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991"
Sep 11 14:54:43.388148 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.388114    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991"} err="failed to get container status \"05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991\": rpc error: code = NotFound desc = could not find container \"05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991\": container with ID starting with 05c4a7be51d948165ae24200ecdc54a003f49ed28eed55cb9a7fa0ff5e702991 not found: ID does not exist"
Sep 11 14:54:43.388148 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.388129    2559 scope.go:117] "RemoveContainer" containerID="cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e"
Sep 11 14:54:43.388402 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:54:43.388382    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e\": container with ID starting with cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e not found: ID does not exist" containerID="cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e"
Sep 11 14:54:43.388491 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.388407    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e"} err="failed to get container status \"cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e\": rpc error: code = NotFound desc = could not find container \"cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e\": container with ID starting with cf98f6efb50abe6908caff331e06f32f669c97dc1ee419c1dfc8f1d35550ff3e not found: ID does not exist"
Sep 11 14:54:43.395280 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.395258    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"]
Sep 11 14:54:43.398071 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.398051    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:43.399587 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.399568    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4408ba1c-gw-openshift-default-7ddf46ddb4-ws7q9"]
Sep 11 14:54:43.594076 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.593998    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:43.722973 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.722939    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:43.870216 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:43.870190    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"]
Sep 11 14:54:43.871708 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:54:43.871683    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9458c3e6_e1cb_4ae3_9eea_b3afef107e8e.slice/crio-6057baabeaf44c18f3c8ebaea5d4726644d7ad31db62a9ebb889215cf3773d7d WatchSource:0}: Error finding container 6057baabeaf44c18f3c8ebaea5d4726644d7ad31db62a9ebb889215cf3773d7d: Status 404 returned error can't find the container with id 6057baabeaf44c18f3c8ebaea5d4726644d7ad31db62a9ebb889215cf3773d7d
Sep 11 14:54:44.180402 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:44.180324    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" path="/var/lib/kubelet/pods/ca4e7c01-392a-4ea8-a24f-9100589e5b70/volumes"
Sep 11 14:54:44.378601 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:44.378562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6" event={"ID":"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e","Type":"ContainerStarted","Data":"ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee"}
Sep 11 14:54:44.378788 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:44.378614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6" event={"ID":"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e","Type":"ContainerStarted","Data":"6057baabeaf44c18f3c8ebaea5d4726644d7ad31db62a9ebb889215cf3773d7d"}
Sep 11 14:54:44.378788 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:44.378646    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:44.398960 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:44.398911    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6" podStartSLOduration=2.064066004 podStartE2EDuration="2.39889809s" totalImagesPullingTime="334.832086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:44.396775559 +0000 UTC m=+890.719403929" watchObservedRunningTime="2026-09-11 14:54:44.39889809 +0000 UTC m=+890.721526459"
Sep 11 14:54:44.542178 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:44.542145    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:45.300715 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:45.300679    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:46.164288 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:46.164244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:46.917305 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:46.917275    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:54:50.388988 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:50.388949    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:54:51.802942 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:51.802913    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:51.803359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:51.803175    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:51.815241 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:51.815214    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:52.428817 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:52.428785    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 14:54:59.413486 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:54:59.413440    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:55:04.813822 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:04.813760    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:05.720225 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:05.720185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:06.414093 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:06.414063    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:07.330804 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:07.330769    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:08.023575 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:08.023542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:08.813751 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:08.813717    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:09.489088 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:09.489042    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:10.189201 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:10.189164    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:10.754230 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:10.754189    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:10.811642 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:10.811600    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:11.038826 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:11.038715    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:11.392688 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:11.392585    2559 patch_prober.go:28] interesting pod/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 14:55:11.392688 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:11.392648    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g" podUID="e24c5901-9e3b-43b6-abd5-52f769464edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 14:55:11.840830 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:11.840793    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:12.404355 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.404327    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"]
Sep 11 14:55:12.404571 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.404548    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6" podUID="9458c3e6-e1cb-4ae3-9eea-b3afef107e8e" containerName="maas-api" containerID="cri-o://ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee" gracePeriod=30
Sep 11 14:55:12.460105 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.460062    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:12.561266 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.561227    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:12.646992 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.646963    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:55:12.660046 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.659975    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:12.761546 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.761507    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:12.784975 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.779754    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w727l\" (UniqueName: \"kubernetes.io/projected/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-kube-api-access-w727l\") pod \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") "
Sep 11 14:55:12.784975 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.779841    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-metrics-certs\") pod \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") "
Sep 11 14:55:12.784975 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.779970    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-maas-api-tls\") pod \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") "
Sep 11 14:55:12.784975 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.779997    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-openshift-service-ca\") pod \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\" (UID: \"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e\") "
Sep 11 14:55:12.784975 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.780736    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9458c3e6-e1cb-4ae3-9eea-b3afef107e8e" (UID: "9458c3e6-e1cb-4ae3-9eea-b3afef107e8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:55:12.786236 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.786176    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9458c3e6-e1cb-4ae3-9eea-b3afef107e8e" (UID: "9458c3e6-e1cb-4ae3-9eea-b3afef107e8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:55:12.786384 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.786313    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-kube-api-access-w727l" (OuterVolumeSpecName: "kube-api-access-w727l") pod "9458c3e6-e1cb-4ae3-9eea-b3afef107e8e" (UID: "9458c3e6-e1cb-4ae3-9eea-b3afef107e8e"). InnerVolumeSpecName "kube-api-access-w727l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:55:12.786952 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.786924    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9458c3e6-e1cb-4ae3-9eea-b3afef107e8e" (UID: "9458c3e6-e1cb-4ae3-9eea-b3afef107e8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:55:12.881815 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.881783    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:12.881815 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.881809    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:12.881815 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.881819    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w727l\" (UniqueName: \"kubernetes.io/projected/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-kube-api-access-w727l\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:12.882421 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:12.881829    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:13.096597 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.096559    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:13.526708 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.526632    2559 generic.go:356] "Generic (PLEG): container finished" podID="9458c3e6-e1cb-4ae3-9eea-b3afef107e8e" containerID="ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee" exitCode=0
Sep 11 14:55:13.526708 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.526677    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"
Sep 11 14:55:13.526957 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.526714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6" event={"ID":"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e","Type":"ContainerDied","Data":"ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee"}
Sep 11 14:55:13.526957 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.526765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6" event={"ID":"9458c3e6-e1cb-4ae3-9eea-b3afef107e8e","Type":"ContainerDied","Data":"6057baabeaf44c18f3c8ebaea5d4726644d7ad31db62a9ebb889215cf3773d7d"}
Sep 11 14:55:13.526957 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.526793    2559 scope.go:117] "RemoveContainer" containerID="ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee"
Sep 11 14:55:13.528512 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.528489    2559 generic.go:356] "Generic (PLEG): container finished" podID="e24c5901-9e3b-43b6-abd5-52f769464edc" containerID="1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb" exitCode=0
Sep 11 14:55:13.528628 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.528517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g" event={"ID":"e24c5901-9e3b-43b6-abd5-52f769464edc","Type":"ContainerDied","Data":"1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb"}
Sep 11 14:55:13.528883 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.528868    2559 scope.go:117] "RemoveContainer" containerID="1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb"
Sep 11 14:55:13.530120 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.530087    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:55:13.530223 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.530176    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:55:13.530287 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.530221    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:55:13.535823 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.535808    2559 scope.go:117] "RemoveContainer" containerID="ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee"
Sep 11 14:55:13.536160 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:55:13.536131    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee\": container with ID starting with ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee not found: ID does not exist" containerID="ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee"
Sep 11 14:55:13.536242 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.536169    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee"} err="failed to get container status \"ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee\": rpc error: code = NotFound desc = could not find container \"ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee\": container with ID starting with ff7402e4562eb837107c55c45a4101ceb59dfca30d454efbea81644f0dba7aee not found: ID does not exist"
Sep 11 14:55:13.576212 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.576175    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"]
Sep 11 14:55:13.578909 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.578885    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7dbbd-694b54dd9-847w6"]
Sep 11 14:55:13.854318 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:13.854267    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:14.106617 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:14.100171    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"]
Sep 11 14:55:14.118042 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:14.118010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:14.182429 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:14.182392    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9458c3e6-e1cb-4ae3-9eea-b3afef107e8e" path="/var/lib/kubelet/pods/9458c3e6-e1cb-4ae3-9eea-b3afef107e8e/volumes"
Sep 11 14:55:14.536038 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:14.536003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g" event={"ID":"e24c5901-9e3b-43b6-abd5-52f769464edc","Type":"ContainerStarted","Data":"ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e"}
Sep 11 14:55:14.536250 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:14.536164    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g" podUID="e24c5901-9e3b-43b6-abd5-52f769464edc" containerName="istio-proxy" containerID="cri-o://ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e" gracePeriod=30
Sep 11 14:55:14.677417 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:14.677377    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:14.793435 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:14.793341    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:15.026719 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.026688    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"]
Sep 11 14:55:15.027087 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.027073    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9458c3e6-e1cb-4ae3-9eea-b3afef107e8e" containerName="maas-api"
Sep 11 14:55:15.027139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.027088    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9458c3e6-e1cb-4ae3-9eea-b3afef107e8e" containerName="maas-api"
Sep 11 14:55:15.027139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.027101    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy"
Sep 11 14:55:15.027139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.027106    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy"
Sep 11 14:55:15.027252 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.027172    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9458c3e6-e1cb-4ae3-9eea-b3afef107e8e" containerName="maas-api"
Sep 11 14:55:15.027252 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.027180    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4e7c01-392a-4ea8-a24f-9100589e5b70" containerName="istio-proxy"
Sep 11 14:55:15.029158 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.029140    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.031619 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.031602    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-58b751e1-openshift-default-dockercfg-pd4sd\""
Sep 11 14:55:15.042596 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.042574    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"]
Sep 11 14:55:15.054586 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.054506    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:15.105108 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.105071    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-envoy\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.105310 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.105110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istiod-ca-cert\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.105310 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.105131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-credential-socket\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.105310 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.105248    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-certs\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.105310 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.105293    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-podinfo\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.105519 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.105337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-socket\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.105519 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.105361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5n7\" (UniqueName: \"kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-kube-api-access-ql5n7\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.105519 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.105470    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-data\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.105519 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.105515    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-token\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.168980 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.168931    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:15.207074 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207034    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-data\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207286 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-token\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207286 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-envoy\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207286 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istiod-ca-cert\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207286 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-credential-socket\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207286 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207257    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-certs\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207591 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207289    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-podinfo\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207591 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-socket\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207591 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207562    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-data\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207757 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207611    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-credential-socket\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207757 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-certs\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207859 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207766    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-socket\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207859 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5n7\" (UniqueName: \"kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-kube-api-access-ql5n7\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.207943 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.207919    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istiod-ca-cert\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.209651 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.209626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-envoy\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.209845 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.209826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-podinfo\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.215641 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.215618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5n7\" (UniqueName: \"kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-kube-api-access-ql5n7\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.215763 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.215742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-token\") pod \"e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") " pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.339381 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.339288    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:15.639420 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.639387    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"]
Sep 11 14:55:15.640740 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:55:15.640716    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8055d0c0_483d_4418_9b00_2ecb0e5ae88f.slice/crio-4950522a4d53dd3a5aceec7b6a1cbfbb0cf0d90ff9e45695f772069a398316b0 WatchSource:0}: Error finding container 4950522a4d53dd3a5aceec7b6a1cbfbb0cf0d90ff9e45695f772069a398316b0: Status 404 returned error can't find the container with id 4950522a4d53dd3a5aceec7b6a1cbfbb0cf0d90ff9e45695f772069a398316b0
Sep 11 14:55:15.643305 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.643262    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:55:15.643382 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.643356    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:55:15.643423 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.643397    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:55:15.717960 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:15.717929    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:16.440959 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:16.440924    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:16.546828 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:16.546797    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" event={"ID":"8055d0c0-483d-4418-9b00-2ecb0e5ae88f","Type":"ContainerStarted","Data":"be28d6137d901e6a9a290101126d73c588727aa6ddf9a2dadbf65dfd3c8e4292"}
Sep 11 14:55:16.547000 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:16.546843    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" event={"ID":"8055d0c0-483d-4418-9b00-2ecb0e5ae88f","Type":"ContainerStarted","Data":"4950522a4d53dd3a5aceec7b6a1cbfbb0cf0d90ff9e45695f772069a398316b0"}
Sep 11 14:55:17.245743 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:17.245703    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:17.339884 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:17.339854    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:17.344644 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:17.344609    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:17.378499 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:17.378410    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" podStartSLOduration=2.378389031 podStartE2EDuration="2.378389031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:55:16.575627649 +0000 UTC m=+922.898256011" watchObservedRunningTime="2026-09-11 14:55:17.378389031 +0000 UTC m=+923.701017402"
Sep 11 14:55:17.552687 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:17.552594    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:17.553562 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:17.553540    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:17.902576 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:17.902492    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:18.008174 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:18.008144    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:18.103369 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:18.103338    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:18.203617 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:18.203529    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:18.951845 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:18.951811    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:19.900280 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:19.900244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:20.611723 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:20.611691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:21.298119 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:21.298080    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:22.117606 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:22.117573    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:22.219225 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:22.219188    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:22.226754 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:22.226728    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:55:22.337357 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:22.337318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:22.745437 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:22.745404    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:22.830901 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:22.830868    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:24.888759 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.888734    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:55:24.998702 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.998676    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-certs\") pod \"e24c5901-9e3b-43b6-abd5-52f769464edc\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") "
Sep 11 14:55:24.998844 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.998711    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-socket\") pod \"e24c5901-9e3b-43b6-abd5-52f769464edc\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") "
Sep 11 14:55:24.998844 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.998750    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24c5901-9e3b-43b6-abd5-52f769464edc-istiod-ca-cert\") pod \"e24c5901-9e3b-43b6-abd5-52f769464edc\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") "
Sep 11 14:55:24.998923 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.998845    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-podinfo\") pod \"e24c5901-9e3b-43b6-abd5-52f769464edc\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") "
Sep 11 14:55:24.998923 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.998870    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-data\") pod \"e24c5901-9e3b-43b6-abd5-52f769464edc\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") "
Sep 11 14:55:24.998923 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.998893    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-envoy\") pod \"e24c5901-9e3b-43b6-abd5-52f769464edc\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") "
Sep 11 14:55:24.999057 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.998935    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqdtd\" (UniqueName: \"kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-kube-api-access-pqdtd\") pod \"e24c5901-9e3b-43b6-abd5-52f769464edc\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") "
Sep 11 14:55:24.999057 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.998975    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-token\") pod \"e24c5901-9e3b-43b6-abd5-52f769464edc\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") "
Sep 11 14:55:24.999057 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.998985    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e24c5901-9e3b-43b6-abd5-52f769464edc" (UID: "e24c5901-9e3b-43b6-abd5-52f769464edc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:55:24.999057 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.999032    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-credential-socket\") pod \"e24c5901-9e3b-43b6-abd5-52f769464edc\" (UID: \"e24c5901-9e3b-43b6-abd5-52f769464edc\") "
Sep 11 14:55:24.999270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.999075    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e24c5901-9e3b-43b6-abd5-52f769464edc" (UID: "e24c5901-9e3b-43b6-abd5-52f769464edc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:55:24.999329 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.999305    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24c5901-9e3b-43b6-abd5-52f769464edc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e24c5901-9e3b-43b6-abd5-52f769464edc" (UID: "e24c5901-9e3b-43b6-abd5-52f769464edc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:55:24.999390 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.999309    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-data" (OuterVolumeSpecName: "istio-data") pod "e24c5901-9e3b-43b6-abd5-52f769464edc" (UID: "e24c5901-9e3b-43b6-abd5-52f769464edc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:55:24.999445 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.999425    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:24.999560 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.999537    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e24c5901-9e3b-43b6-abd5-52f769464edc" (UID: "e24c5901-9e3b-43b6-abd5-52f769464edc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:55:24.999615 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.999558    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-workload-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:24.999615 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:24.999580    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24c5901-9e3b-43b6-abd5-52f769464edc-istiod-ca-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:25.001124 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.001090    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-token" (OuterVolumeSpecName: "istio-token") pod "e24c5901-9e3b-43b6-abd5-52f769464edc" (UID: "e24c5901-9e3b-43b6-abd5-52f769464edc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:55:25.001229 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.001129    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e24c5901-9e3b-43b6-abd5-52f769464edc" (UID: "e24c5901-9e3b-43b6-abd5-52f769464edc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:55:25.001229 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.001178    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e24c5901-9e3b-43b6-abd5-52f769464edc" (UID: "e24c5901-9e3b-43b6-abd5-52f769464edc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:55:25.001229 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.001193    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-kube-api-access-pqdtd" (OuterVolumeSpecName: "kube-api-access-pqdtd") pod "e24c5901-9e3b-43b6-abd5-52f769464edc" (UID: "e24c5901-9e3b-43b6-abd5-52f769464edc"). InnerVolumeSpecName "kube-api-access-pqdtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:55:25.100767 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.100737    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-podinfo\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:25.100767 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.100763    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-data\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:25.100767 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.100774    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-envoy\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:25.100982 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.100786    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqdtd\" (UniqueName: \"kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-kube-api-access-pqdtd\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:25.100982 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.100798    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24c5901-9e3b-43b6-abd5-52f769464edc-istio-token\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:25.100982 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.100809    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24c5901-9e3b-43b6-abd5-52f769464edc-credential-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:55:25.598575 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.598539    2559 generic.go:356] "Generic (PLEG): container finished" podID="e24c5901-9e3b-43b6-abd5-52f769464edc" containerID="ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e" exitCode=0
Sep 11 14:55:25.598779 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.598618    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"
Sep 11 14:55:25.598779 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.598660    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g" event={"ID":"e24c5901-9e3b-43b6-abd5-52f769464edc","Type":"ContainerDied","Data":"ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e"}
Sep 11 14:55:25.598779 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.598737    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g" event={"ID":"e24c5901-9e3b-43b6-abd5-52f769464edc","Type":"ContainerDied","Data":"d9e7bca2a64f4405240677086089992485cf3ee88e681182b0a184209e1df8f8"}
Sep 11 14:55:25.598935 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.598805    2559 scope.go:117] "RemoveContainer" containerID="ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e"
Sep 11 14:55:25.612661 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.612644    2559 scope.go:117] "RemoveContainer" containerID="1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb"
Sep 11 14:55:25.621056 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.621037    2559 scope.go:117] "RemoveContainer" containerID="ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e"
Sep 11 14:55:25.621313 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:55:25.621293    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e\": container with ID starting with ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e not found: ID does not exist" containerID="ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e"
Sep 11 14:55:25.621368 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.621327    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e"} err="failed to get container status \"ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e\": rpc error: code = NotFound desc = could not find container \"ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e\": container with ID starting with ae20cfb609994d28128ba6e16ec14094a46d246b5e5939f6aa988e2e89dbda7e not found: ID does not exist"
Sep 11 14:55:25.621368 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.621347    2559 scope.go:117] "RemoveContainer" containerID="1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb"
Sep 11 14:55:25.621716 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:55:25.621694    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb\": container with ID starting with 1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb not found: ID does not exist" containerID="1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb"
Sep 11 14:55:25.621811 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.621720    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb"} err="failed to get container status \"1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb\": rpc error: code = NotFound desc = could not find container \"1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb\": container with ID starting with 1d215689931fbf46a816f9d7949975d8d3c3b6d037fbbce0f307f38b10bf99eb not found: ID does not exist"
Sep 11 14:55:25.629892 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.629870    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"]
Sep 11 14:55:25.634141 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:25.634122    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c0b7dbbd-openshift-default-55f9585ddd-6z62g"]
Sep 11 14:55:26.179821 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:26.179789    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e24c5901-9e3b-43b6-abd5-52f769464edc" path="/var/lib/kubelet/pods/e24c5901-9e3b-43b6-abd5-52f769464edc/volumes"
Sep 11 14:55:28.962814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:28.962772    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"]
Sep 11 14:55:28.963329 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:28.963310    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e24c5901-9e3b-43b6-abd5-52f769464edc" containerName="istio-proxy"
Sep 11 14:55:28.963329 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:28.963329    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e24c5901-9e3b-43b6-abd5-52f769464edc" containerName="istio-proxy"
Sep 11 14:55:28.963510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:28.963413    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e24c5901-9e3b-43b6-abd5-52f769464edc" containerName="istio-proxy"
Sep 11 14:55:28.963510 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:28.963422    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e24c5901-9e3b-43b6-abd5-52f769464edc" containerName="istio-proxy"
Sep 11 14:55:28.968201 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:28.968178    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:28.971179 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:28.971152    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3219e3\"/\"kube-root-ca.crt\""
Sep 11 14:55:28.972330 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:28.972308    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3219e3\"/\"autores-model-3219e3-kserve-self-signed-certs\""
Sep 11 14:55:28.972448 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:28.972340    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3219e3\"/\"default-dockercfg-gq5sd\""
Sep 11 14:55:28.972448 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:28.972350    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3219e3\"/\"openshift-service-ca.crt\""
Sep 11 14:55:28.983927 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:28.983905    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"]
Sep 11 14:55:29.000290 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.000261    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:29.031805 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.031783    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-home\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.031930 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.031824    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-dshm\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.031930 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.031845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tmp-dir\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.031930 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.031917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4s2\" (UniqueName: \"kubernetes.io/projected/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-kube-api-access-7v4s2\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.032093 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.031974    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-model-cache\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.032093 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.032013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tls-certs\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.132511 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.132478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-home\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.132690 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.132547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-dshm\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.132690 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.132571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tmp-dir\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.132690 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.132611    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4s2\" (UniqueName: \"kubernetes.io/projected/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-kube-api-access-7v4s2\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.132690 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.132656    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-model-cache\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.132914 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.132694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tls-certs\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.132914 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.132907    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-home\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.133036 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.133017    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tmp-dir\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.133078 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.133066    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-model-cache\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.134820 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.134799    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-dshm\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.135058 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.135039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tls-certs\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.140914 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.140891    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4s2\" (UniqueName: \"kubernetes.io/projected/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-kube-api-access-7v4s2\") pod \"autores-model-3219e3-kserve-66d47c87cf-9rr6b\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") " pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.278257 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.278228    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:29.550224 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.550049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"]
Sep 11 14:55:29.553031 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:55:29.553006    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cc605dc_7b70_4408_85ca_ac54bb0dc2f7.slice/crio-3c486e7f4061143a742fcc299bede6d2c132d430c6e60b9972ab507a60347fb4 WatchSource:0}: Error finding container 3c486e7f4061143a742fcc299bede6d2c132d430c6e60b9972ab507a60347fb4: Status 404 returned error can't find the container with id 3c486e7f4061143a742fcc299bede6d2c132d430c6e60b9972ab507a60347fb4
Sep 11 14:55:29.621215 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.621188    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b" event={"ID":"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7","Type":"ContainerStarted","Data":"2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6"}
Sep 11 14:55:29.621341 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.621225    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b" event={"ID":"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7","Type":"ContainerStarted","Data":"3c486e7f4061143a742fcc299bede6d2c132d430c6e60b9972ab507a60347fb4"}
Sep 11 14:55:29.640266 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:29.640222    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b" podStartSLOduration=1.640206213 podStartE2EDuration="1.640206213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:55:29.639126287 +0000 UTC m=+935.961754656" watchObservedRunningTime="2026-09-11 14:55:29.640206213 +0000 UTC m=+935.962834582"
Sep 11 14:55:39.278701 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:39.278655    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:39.279208 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:39.278742    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:39.293665 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:39.293635    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:39.694196 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:39.694118    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:55:41.373037 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.372999    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"]
Sep 11 14:55:41.373585 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.373569    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e24c5901-9e3b-43b6-abd5-52f769464edc" containerName="istio-proxy"
Sep 11 14:55:41.373630 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.373587    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e24c5901-9e3b-43b6-abd5-52f769464edc" containerName="istio-proxy"
Sep 11 14:55:41.377193 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.377173    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.379910 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.379882    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-362f7a\"/\"kube-root-ca.crt\""
Sep 11 14:55:41.380019 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.379922    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-362f7a\"/\"test-model-362f7a-kserve-self-signed-certs\""
Sep 11 14:55:41.380106 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.380011    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-362f7a\"/\"openshift-service-ca.crt\""
Sep 11 14:55:41.380983 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.380966    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-362f7a\"/\"default-dockercfg-9frxb\""
Sep 11 14:55:41.395764 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.389136    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"]
Sep 11 14:55:41.404701 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.404668    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:41.442551 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.442519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-home\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.442706 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.442614    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-dshm\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.442706 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.442650    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-tmp-dir\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.442706 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.442672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-model-cache\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.442706 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.442688    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f0f5ed-f848-431f-82c0-123bfcc68d58-tls-certs\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.442706 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.442705    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlk5s\" (UniqueName: \"kubernetes.io/projected/40f0f5ed-f848-431f-82c0-123bfcc68d58-kube-api-access-hlk5s\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.543536 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.543499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-model-cache\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.543536 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.543539    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f0f5ed-f848-431f-82c0-123bfcc68d58-tls-certs\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.543783 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.543559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlk5s\" (UniqueName: \"kubernetes.io/projected/40f0f5ed-f848-431f-82c0-123bfcc68d58-kube-api-access-hlk5s\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.543783 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.543596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-home\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.543783 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.543682    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-dshm\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.543783 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.543722    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-tmp-dir\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.544007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.543963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-model-cache\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.544007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.543998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-home\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.544125 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.544029    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-tmp-dir\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.545947 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.545916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-dshm\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.546130 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.546115    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f0f5ed-f848-431f-82c0-123bfcc68d58-tls-certs\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.552392 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.552363    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlk5s\" (UniqueName: \"kubernetes.io/projected/40f0f5ed-f848-431f-82c0-123bfcc68d58-kube-api-access-hlk5s\") pod \"test-model-362f7a-kserve-76884c6587-w7cbm\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") " pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.689863 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.689769    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:41.962141 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:41.962101    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"]
Sep 11 14:55:41.967520 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:55:41.967489    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40f0f5ed_f848_431f_82c0_123bfcc68d58.slice/crio-b538f79beee2bb772ac0fb7b8fe4a9139a1368d5be0a6337251660a222e8ab92 WatchSource:0}: Error finding container b538f79beee2bb772ac0fb7b8fe4a9139a1368d5be0a6337251660a222e8ab92: Status 404 returned error can't find the container with id b538f79beee2bb772ac0fb7b8fe4a9139a1368d5be0a6337251660a222e8ab92
Sep 11 14:55:42.696681 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:42.696647    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm" event={"ID":"40f0f5ed-f848-431f-82c0-123bfcc68d58","Type":"ContainerStarted","Data":"fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5"}
Sep 11 14:55:42.697046 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:42.696700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm" event={"ID":"40f0f5ed-f848-431f-82c0-123bfcc68d58","Type":"ContainerStarted","Data":"b538f79beee2bb772ac0fb7b8fe4a9139a1368d5be0a6337251660a222e8ab92"}
Sep 11 14:55:42.718677 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:42.718627    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm" podStartSLOduration=1.718610495 podStartE2EDuration="1.718610495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:55:42.716687663 +0000 UTC m=+949.039316029" watchObservedRunningTime="2026-09-11 14:55:42.718610495 +0000 UTC m=+949.041238864"
Sep 11 14:55:43.703393 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:43.703360    2559 generic.go:356] "Generic (PLEG): container finished" podID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerID="be28d6137d901e6a9a290101126d73c588727aa6ddf9a2dadbf65dfd3c8e4292" exitCode=0
Sep 11 14:55:43.703872 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:43.703431    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" event={"ID":"8055d0c0-483d-4418-9b00-2ecb0e5ae88f","Type":"ContainerDied","Data":"be28d6137d901e6a9a290101126d73c588727aa6ddf9a2dadbf65dfd3c8e4292"}
Sep 11 14:55:43.703996 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:43.703980    2559 scope.go:117] "RemoveContainer" containerID="be28d6137d901e6a9a290101126d73c588727aa6ddf9a2dadbf65dfd3c8e4292"
Sep 11 14:55:43.705021 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:43.704997    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:55:43.705079 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:43.705060    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:55:43.705116 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:43.705089    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:55:44.383290 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:44.383245    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:44.710857 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:44.710762    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" event={"ID":"8055d0c0-483d-4418-9b00-2ecb0e5ae88f","Type":"ContainerStarted","Data":"537186ce974f800c57b61f75bbc326b3900c1d475daa6dfb02e68207347ce9e3"}
Sep 11 14:55:45.103340 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:45.103307    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:45.340313 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:45.340276    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:45.341324 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:45.341297    2559 patch_prober.go:28] interesting pod/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 11 14:55:45.341442 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:45.341346    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 11 14:55:45.831622 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:45.831580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:46.340711 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:46.340674    2559 patch_prober.go:28] interesting pod/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 11 14:55:46.340894 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:46.340747    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 11 14:55:46.597524 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:46.597411    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:47.287717 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:47.287665    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:47.339624 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:47.339588    2559 patch_prober.go:28] interesting pod/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 11 14:55:47.339784 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:47.339650    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 11 14:55:47.553501 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:47.553399    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:48.112783 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:48.112746    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:48.339863 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:48.339827    2559 patch_prober.go:28] interesting pod/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 11 14:55:48.340241 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:48.339884    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 11 14:55:48.820363 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:48.820329    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:49.339869 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:49.339838    2559 patch_prober.go:28] interesting pod/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 11 14:55:49.340249 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:49.339893    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 11 14:55:49.478945 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:49.478904    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:50.149353 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:50.149317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:50.344603 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:50.344575    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:50.345643 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:50.345626    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:55:50.848894 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:50.848858    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:50.930564 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:50.930527    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:55:51.690281 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:51.690243    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:51.690682 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:51.690294    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:51.703111 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:51.703085    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:51.757390 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:51.757361    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:55:51.935737 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:51.935694    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:52.984813 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:52.984778    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:53.834245 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:53.834212    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:55:59.620309 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:55:59.620275    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:00.229808 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:00.229775    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:00.297386 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:00.297350    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:00.367768 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:00.367726    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:00.431175 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:00.431148    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:00.564668 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:00.564561    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"]
Sep 11 14:56:00.564847 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:00.564790    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy" containerID="cri-o://537186ce974f800c57b61f75bbc326b3900c1d475daa6dfb02e68207347ce9e3" gracePeriod=30
Sep 11 14:56:00.575476 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:00.575426    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:00.896341 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:00.896252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:01.264518 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:01.264445    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:01.359883 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:01.359843    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:01.468139 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:01.468106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:02.170328 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:02.170289    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:02.883817 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:02.883764    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:02.980814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:02.980782    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:03.705726 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:03.705687    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:04.441870 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:04.441820    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:05.170928 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.170895    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:05.347438 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.347399    2559 patch_prober.go:28] interesting pod/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 14:56:05.347594 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.347474    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 14:56:05.818039 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.818001    2559 generic.go:356] "Generic (PLEG): container finished" podID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerID="537186ce974f800c57b61f75bbc326b3900c1d475daa6dfb02e68207347ce9e3" exitCode=0
Sep 11 14:56:05.818205 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.818063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" event={"ID":"8055d0c0-483d-4418-9b00-2ecb0e5ae88f","Type":"ContainerDied","Data":"537186ce974f800c57b61f75bbc326b3900c1d475daa6dfb02e68207347ce9e3"}
Sep 11 14:56:05.818205 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.818120    2559 scope.go:117] "RemoveContainer" containerID="be28d6137d901e6a9a290101126d73c588727aa6ddf9a2dadbf65dfd3c8e4292"
Sep 11 14:56:05.914330 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.914298    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:56:05.942824 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.942745    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:05.982774 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.982745    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istiod-ca-cert\") pod \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") "
Sep 11 14:56:05.982933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.982785    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-podinfo\") pod \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") "
Sep 11 14:56:05.982933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.982837    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-certs\") pod \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") "
Sep 11 14:56:05.982933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.982858    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-token\") pod \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") "
Sep 11 14:56:05.982933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.982882    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-envoy\") pod \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") "
Sep 11 14:56:05.982933 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.982923    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-data\") pod \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") "
Sep 11 14:56:05.983183 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.982970    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-socket\") pod \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") "
Sep 11 14:56:05.983183 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.983043    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql5n7\" (UniqueName: \"kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-kube-api-access-ql5n7\") pod \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") "
Sep 11 14:56:05.983183 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.983092    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-credential-socket\") pod \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\" (UID: \"8055d0c0-483d-4418-9b00-2ecb0e5ae88f\") "
Sep 11 14:56:05.983183 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.983134    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8055d0c0-483d-4418-9b00-2ecb0e5ae88f" (UID: "8055d0c0-483d-4418-9b00-2ecb0e5ae88f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:56:05.983183 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.983087    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8055d0c0-483d-4418-9b00-2ecb0e5ae88f" (UID: "8055d0c0-483d-4418-9b00-2ecb0e5ae88f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:56:05.983492 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.983289    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8055d0c0-483d-4418-9b00-2ecb0e5ae88f" (UID: "8055d0c0-483d-4418-9b00-2ecb0e5ae88f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:56:05.983492 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.983372    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-data" (OuterVolumeSpecName: "istio-data") pod "8055d0c0-483d-4418-9b00-2ecb0e5ae88f" (UID: "8055d0c0-483d-4418-9b00-2ecb0e5ae88f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:56:05.983492 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.983481    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8055d0c0-483d-4418-9b00-2ecb0e5ae88f" (UID: "8055d0c0-483d-4418-9b00-2ecb0e5ae88f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:56:05.983683 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.983555    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-data\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:05.983683 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.983569    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:05.983683 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.983578    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istiod-ca-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:05.983683 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.983588    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-workload-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:05.985048 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.985026    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8055d0c0-483d-4418-9b00-2ecb0e5ae88f" (UID: "8055d0c0-483d-4418-9b00-2ecb0e5ae88f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:56:05.985048 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.985034    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8055d0c0-483d-4418-9b00-2ecb0e5ae88f" (UID: "8055d0c0-483d-4418-9b00-2ecb0e5ae88f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:56:05.985207 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.985034    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-token" (OuterVolumeSpecName: "istio-token") pod "8055d0c0-483d-4418-9b00-2ecb0e5ae88f" (UID: "8055d0c0-483d-4418-9b00-2ecb0e5ae88f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:56:05.985296 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:05.985269    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-kube-api-access-ql5n7" (OuterVolumeSpecName: "kube-api-access-ql5n7") pod "8055d0c0-483d-4418-9b00-2ecb0e5ae88f" (UID: "8055d0c0-483d-4418-9b00-2ecb0e5ae88f"). InnerVolumeSpecName "kube-api-access-ql5n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:56:06.084615 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:06.084583    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-envoy\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:06.084615 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:06.084611    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql5n7\" (UniqueName: \"kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-kube-api-access-ql5n7\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:06.084615 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:06.084622    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-credential-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:06.084833 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:06.084633    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-podinfo\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:06.084833 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:06.084647    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8055d0c0-483d-4418-9b00-2ecb0e5ae88f-istio-token\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:06.824701 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:06.824671    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"
Sep 11 14:56:06.825195 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:06.824710    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h" event={"ID":"8055d0c0-483d-4418-9b00-2ecb0e5ae88f","Type":"ContainerDied","Data":"4950522a4d53dd3a5aceec7b6a1cbfbb0cf0d90ff9e45695f772069a398316b0"}
Sep 11 14:56:06.825195 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:06.824767    2559 scope.go:117] "RemoveContainer" containerID="537186ce974f800c57b61f75bbc326b3900c1d475daa6dfb02e68207347ce9e3"
Sep 11 14:56:06.844491 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:06.844449    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"]
Sep 11 14:56:06.848747 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:06.848722    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-58b751e1-openshift-default-67b877b996-zvx8h"]
Sep 11 14:56:08.181059 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:08.181020    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" path="/var/lib/kubelet/pods/8055d0c0-483d-4418-9b00-2ecb0e5ae88f/volumes"
Sep 11 14:56:09.062125 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:09.062084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:09.129197 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:09.129165    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:09.231180 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:09.231142    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:09.338137 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:09.338052    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:09.460929 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:09.460887    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:09.904444 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:09.904404    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:11.876867 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:11.876834    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:12.471134 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.471095    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"]
Sep 11 14:56:12.471414 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.471387    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn" podUID="f7f07d25-e503-4fae-a9c0-63ac52053758" containerName="maas-api" containerID="cri-o://3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958" gracePeriod=30
Sep 11 14:56:12.502390 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.502362    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:12.575254 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.575213    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:12.639643 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.639576    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:12.712000 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.711961    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:12.722208 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.722149    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:56:12.795842 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.795812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:12.850323 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.850287    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-metrics-certs\") pod \"f7f07d25-e503-4fae-a9c0-63ac52053758\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") "
Sep 11 14:56:12.850504 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.850350    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-maas-api-tls\") pod \"f7f07d25-e503-4fae-a9c0-63ac52053758\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") "
Sep 11 14:56:12.850504 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.850425    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f07d25-e503-4fae-a9c0-63ac52053758-openshift-service-ca\") pod \"f7f07d25-e503-4fae-a9c0-63ac52053758\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") "
Sep 11 14:56:12.850631 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.850544    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f89g5\" (UniqueName: \"kubernetes.io/projected/f7f07d25-e503-4fae-a9c0-63ac52053758-kube-api-access-f89g5\") pod \"f7f07d25-e503-4fae-a9c0-63ac52053758\" (UID: \"f7f07d25-e503-4fae-a9c0-63ac52053758\") "
Sep 11 14:56:12.850942 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.850907    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f07d25-e503-4fae-a9c0-63ac52053758-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7f07d25-e503-4fae-a9c0-63ac52053758" (UID: "f7f07d25-e503-4fae-a9c0-63ac52053758"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:56:12.852712 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.852681    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f7f07d25-e503-4fae-a9c0-63ac52053758" (UID: "f7f07d25-e503-4fae-a9c0-63ac52053758"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:56:12.852712 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.852702    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7f07d25-e503-4fae-a9c0-63ac52053758" (UID: "f7f07d25-e503-4fae-a9c0-63ac52053758"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:56:12.852935 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.852910    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f07d25-e503-4fae-a9c0-63ac52053758-kube-api-access-f89g5" (OuterVolumeSpecName: "kube-api-access-f89g5") pod "f7f07d25-e503-4fae-a9c0-63ac52053758" (UID: "f7f07d25-e503-4fae-a9c0-63ac52053758"). InnerVolumeSpecName "kube-api-access-f89g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:56:12.856742 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.856721    2559 generic.go:356] "Generic (PLEG): container finished" podID="f7f07d25-e503-4fae-a9c0-63ac52053758" containerID="3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958" exitCode=0
Sep 11 14:56:12.856860 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.856778    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"
Sep 11 14:56:12.856860 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.856803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn" event={"ID":"f7f07d25-e503-4fae-a9c0-63ac52053758","Type":"ContainerDied","Data":"3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958"}
Sep 11 14:56:12.856972 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.856869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn" event={"ID":"f7f07d25-e503-4fae-a9c0-63ac52053758","Type":"ContainerDied","Data":"0494fc8bdad551030c902db43fadc12a204c0bd3de63cbb3b32c0c41257af302"}
Sep 11 14:56:12.856972 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.856902    2559 scope.go:117] "RemoveContainer" containerID="3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958"
Sep 11 14:56:12.872427 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.872402    2559 scope.go:117] "RemoveContainer" containerID="3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958"
Sep 11 14:56:12.872732 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:56:12.872712    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958\": container with ID starting with 3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958 not found: ID does not exist" containerID="3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958"
Sep 11 14:56:12.872800 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.872746    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958"} err="failed to get container status \"3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958\": rpc error: code = NotFound desc = could not find container \"3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958\": container with ID starting with 3d909620221340514b29ef2307b8746abf19f9b044a16d16495c71c8c811f958 not found: ID does not exist"
Sep 11 14:56:12.884887 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.884854    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"]
Sep 11 14:56:12.886600 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.886582    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3219e3-754df69985-w79cn"]
Sep 11 14:56:12.951796 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.951770    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f89g5\" (UniqueName: \"kubernetes.io/projected/f7f07d25-e503-4fae-a9c0-63ac52053758-kube-api-access-f89g5\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:12.951796 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.951792    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:12.951931 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.951804    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f07d25-e503-4fae-a9c0-63ac52053758-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:12.951931 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:12.951812    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f07d25-e503-4fae-a9c0-63ac52053758-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:13.117807 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:13.117701    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:13.217797 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:13.217766    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:13.907791 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:13.907756    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:14.181535 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:14.181433    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7f07d25-e503-4fae-a9c0-63ac52053758" path="/var/lib/kubelet/pods/f7f07d25-e503-4fae-a9c0-63ac52053758/volumes"
Sep 11 14:56:14.774892 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:14.774852    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:15.932344 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:15.932308    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:16.420783 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:16.420684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:17.225173 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:17.225136    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:56:18.093492 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:18.093442    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"]
Sep 11 14:56:18.189835 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:18.189802    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:19.172941 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:19.172905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:21.218559 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:21.218516    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:33.112828 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.112785    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b" podUID="1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" containerName="main" containerID="cri-o://2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6" gracePeriod=45
Sep 11 14:56:33.363247 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.363217    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:56:33.447033 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447004    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-model-cache\") pod \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") "
Sep 11 14:56:33.447215 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447058    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tmp-dir\") pod \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") "
Sep 11 14:56:33.447215 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447083    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tls-certs\") pod \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") "
Sep 11 14:56:33.447215 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447111    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-dshm\") pod \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") "
Sep 11 14:56:33.447215 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447165    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-home\") pod \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") "
Sep 11 14:56:33.447444 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447217    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v4s2\" (UniqueName: \"kubernetes.io/projected/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-kube-api-access-7v4s2\") pod \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\" (UID: \"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7\") "
Sep 11 14:56:33.447444 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447340    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" (UID: "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:56:33.447444 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447378    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" (UID: "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:56:33.447634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447474    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-home" (OuterVolumeSpecName: "home") pod "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" (UID: "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:56:33.447634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447626    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:33.447744 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447641    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:33.447744 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.447655    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:33.449262 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.449229    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-dshm" (OuterVolumeSpecName: "dshm") pod "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" (UID: "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:56:33.449368 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.449306    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" (UID: "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:56:33.449368 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.449317    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-kube-api-access-7v4s2" (OuterVolumeSpecName: "kube-api-access-7v4s2") pod "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" (UID: "1cc605dc-7b70-4408-85ca-ac54bb0dc2f7"). InnerVolumeSpecName "kube-api-access-7v4s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:56:33.548345 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.548267    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:33.548345 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.548293    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:33.548345 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.548304    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v4s2\" (UniqueName: \"kubernetes.io/projected/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7-kube-api-access-7v4s2\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:56:33.963557 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.963435    2559 generic.go:356] "Generic (PLEG): container finished" podID="1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" containerID="2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6" exitCode=0
Sep 11 14:56:33.963557 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.963498    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b" event={"ID":"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7","Type":"ContainerDied","Data":"2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6"}
Sep 11 14:56:33.963557 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.963548    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b" event={"ID":"1cc605dc-7b70-4408-85ca-ac54bb0dc2f7","Type":"ContainerDied","Data":"3c486e7f4061143a742fcc299bede6d2c132d430c6e60b9972ab507a60347fb4"}
Sep 11 14:56:33.963773 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.963566    2559 scope.go:117] "RemoveContainer" containerID="2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6"
Sep 11 14:56:33.963773 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.963506    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"
Sep 11 14:56:33.972584 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.972563    2559 scope.go:117] "RemoveContainer" containerID="2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6"
Sep 11 14:56:33.972845 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:56:33.972826    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6\": container with ID starting with 2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6 not found: ID does not exist" containerID="2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6"
Sep 11 14:56:33.972908 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.972851    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6"} err="failed to get container status \"2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6\": rpc error: code = NotFound desc = could not find container \"2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6\": container with ID starting with 2670912b991e76d212d2eb5bae6bc2868078a2f95ccd8522c19cc983eab5daf6 not found: ID does not exist"
Sep 11 14:56:33.990555 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.990524    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"]
Sep 11 14:56:33.995041 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:33.995019    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-3219e3/autores-model-3219e3-kserve-66d47c87cf-9rr6b"]
Sep 11 14:56:34.181992 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:34.181962    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" path="/var/lib/kubelet/pods/1cc605dc-7b70-4408-85ca-ac54bb0dc2f7/volumes"
Sep 11 14:56:35.950217 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:35.950181    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:41.205384 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:41.205331    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:42.337846 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:42.337798    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:42.441137 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:42.441104    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:42.556811 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:42.556775    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:46.085792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.085757    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"]
Sep 11 14:56:46.086189 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.086160    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" containerName="main"
Sep 11 14:56:46.086189 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.086171    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" containerName="main"
Sep 11 14:56:46.086189 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.086185    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy"
Sep 11 14:56:46.086189 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.086192    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy"
Sep 11 14:56:46.086404 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.086200    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f07d25-e503-4fae-a9c0-63ac52053758" containerName="maas-api"
Sep 11 14:56:46.086404 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.086207    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f07d25-e503-4fae-a9c0-63ac52053758" containerName="maas-api"
Sep 11 14:56:46.086404 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.086276    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f07d25-e503-4fae-a9c0-63ac52053758" containerName="maas-api"
Sep 11 14:56:46.086404 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.086282    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy"
Sep 11 14:56:46.086404 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.086289    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc605dc-7b70-4408-85ca-ac54bb0dc2f7" containerName="main"
Sep 11 14:56:46.086404 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.086295    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy"
Sep 11 14:56:46.090877 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.090848    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.094161 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.094129    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"premium-w0-e7ca56-kserve-self-signed-certs\""
Sep 11 14:56:46.103949 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.103920    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"]
Sep 11 14:56:46.119032 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.118998    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:46.174368 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.174332    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-home\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.174582 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.174400    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-tmp-dir\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.174582 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.174515    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-model-cache\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.174582 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.174554    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzfc6\" (UniqueName: \"kubernetes.io/projected/b5b2b453-4c05-4a53-bedf-319e648831fd-kube-api-access-pzfc6\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.174771 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.174744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-dshm\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.174831 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.174785    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b2b453-4c05-4a53-bedf-319e648831fd-tls-certs\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.275283 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.275241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-dshm\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.275283 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.275283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b2b453-4c05-4a53-bedf-319e648831fd-tls-certs\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.275563 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.275418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-home\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.275563 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.275545    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-tmp-dir\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.275669 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.275592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-model-cache\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.275669 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.275622    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzfc6\" (UniqueName: \"kubernetes.io/projected/b5b2b453-4c05-4a53-bedf-319e648831fd-kube-api-access-pzfc6\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.275983 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.275936    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-tmp-dir\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.276112 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.275996    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-model-cache\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.276324 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.276298    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-home\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.277674 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.277652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-dshm\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.277975 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.277959    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b2b453-4c05-4a53-bedf-319e648831fd-tls-certs\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.285041 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.285016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzfc6\" (UniqueName: \"kubernetes.io/projected/b5b2b453-4c05-4a53-bedf-319e648831fd-kube-api-access-pzfc6\") pod \"premium-w0-e7ca56-kserve-65f5459c4f-rgdjf\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") " pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.401141 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.401048    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:46.677194 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.677161    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"]
Sep 11 14:56:46.721773 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:46.721740    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:56:47.031475 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:47.031408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf" event={"ID":"b5b2b453-4c05-4a53-bedf-319e648831fd","Type":"ContainerStarted","Data":"3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4"}
Sep 11 14:56:47.031669 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:47.031482    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf" event={"ID":"b5b2b453-4c05-4a53-bedf-319e648831fd","Type":"ContainerStarted","Data":"32dfd74449332b6a7ae20340a3445d9e6b9125b59b48cdae2a9a475a57096b3e"}
Sep 11 14:56:47.050631 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:47.050572    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf" podStartSLOduration=1.050553218 podStartE2EDuration="1.050553218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:56:47.048592905 +0000 UTC m=+1013.371221276" watchObservedRunningTime="2026-09-11 14:56:47.050553218 +0000 UTC m=+1013.373181616"
Sep 11 14:56:56.401875 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.401830    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:56.402416 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.401894    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:56.415298 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.415227    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:56.592750 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.592714    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"]
Sep 11 14:56:56.593177 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.593157    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy"
Sep 11 14:56:56.593177 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.593175    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8055d0c0-483d-4418-9b00-2ecb0e5ae88f" containerName="istio-proxy"
Sep 11 14:56:56.597556 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.597538    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.600357 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.600339    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-90d6f5\"/\"kube-root-ca.crt\""
Sep 11 14:56:56.600357 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.600345    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-90d6f5\"/\"default-dockercfg-77hc8\""
Sep 11 14:56:56.600554 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.600368    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-90d6f5\"/\"test-model-90d6f5-kserve-self-signed-certs\""
Sep 11 14:56:56.600670 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.600655    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-90d6f5\"/\"openshift-service-ca.crt\""
Sep 11 14:56:56.607921 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.607899    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"]
Sep 11 14:56:56.633320 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.633288    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:56:56.782177 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.782138    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgltk\" (UniqueName: \"kubernetes.io/projected/598c5590-02d6-41ec-8e30-12043837e367-kube-api-access-tgltk\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.782382 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.782256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-home\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.782382 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.782297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-tmp-dir\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.782382 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.782341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-model-cache\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.782555 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.782401    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-dshm\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.782555 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.782445    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598c5590-02d6-41ec-8e30-12043837e367-tls-certs\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.882980 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.882941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-dshm\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.883155 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.882986    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598c5590-02d6-41ec-8e30-12043837e367-tls-certs\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.883155 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.883112    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgltk\" (UniqueName: \"kubernetes.io/projected/598c5590-02d6-41ec-8e30-12043837e367-kube-api-access-tgltk\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.883296 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.883276    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-home\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.883353 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.883309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-tmp-dir\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.883405 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.883360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-model-cache\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.883649 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.883622    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-home\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.883777 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.883676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-tmp-dir\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.883777 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.883765    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-model-cache\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.885362 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.885331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-dshm\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.885541 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.885523    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598c5590-02d6-41ec-8e30-12043837e367-tls-certs\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.894494 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.894449    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgltk\" (UniqueName: \"kubernetes.io/projected/598c5590-02d6-41ec-8e30-12043837e367-kube-api-access-tgltk\") pod \"test-model-90d6f5-kserve-b5df94f69-tltlj\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") " pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:56.907704 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:56.907682    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:56:57.095138 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:57.095045    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 14:56:57.187264 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:57.187242    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"]
Sep 11 14:56:57.190008 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:56:57.189984    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod598c5590_02d6_41ec_8e30_12043837e367.slice/crio-0f42dc93e8c8e66ff4ed407084672f42f36c2c0dd753ad215ab67c5f06aaa241 WatchSource:0}: Error finding container 0f42dc93e8c8e66ff4ed407084672f42f36c2c0dd753ad215ab67c5f06aaa241: Status 404 returned error can't find the container with id 0f42dc93e8c8e66ff4ed407084672f42f36c2c0dd753ad215ab67c5f06aaa241
Sep 11 14:56:58.096733 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:58.096689    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj" event={"ID":"598c5590-02d6-41ec-8e30-12043837e367","Type":"ContainerStarted","Data":"18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8"}
Sep 11 14:56:58.097136 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:58.096746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj" event={"ID":"598c5590-02d6-41ec-8e30-12043837e367","Type":"ContainerStarted","Data":"0f42dc93e8c8e66ff4ed407084672f42f36c2c0dd753ad215ab67c5f06aaa241"}
Sep 11 14:56:58.116035 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:58.115975    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj" podStartSLOduration=2.115958613 podStartE2EDuration="2.115958613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:56:58.11492751 +0000 UTC m=+1024.437555879" watchObservedRunningTime="2026-09-11 14:56:58.115958613 +0000 UTC m=+1024.438586982"
Sep 11 14:56:58.464574 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:56:58.464482    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:03.328797 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:03.328759    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:57:05.574627 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:05.574583    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:05.831582 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:05.831494    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:06.350238 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.350202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:06.459966 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.459925    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:06.576930 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.576892    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:06.664298 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.664221    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"]
Sep 11 14:57:06.667414 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.667391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.670889 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.670864    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 14:57:06.671651 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.671636    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-87d086-openshift-default-dockercfg-s28jd\""
Sep 11 14:57:06.685088 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.685066    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"]
Sep 11 14:57:06.694818 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.694792    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:06.778371 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.778338    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-data\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.778576 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.778379    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-credential-socket\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.778576 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.778398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-podinfo\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.778576 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.778487    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-envoy\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.778576 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.778528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istiod-ca-cert\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.778576 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.778565    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-token\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.778751 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.778640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-certs\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.778751 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.778676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zgk\" (UniqueName: \"kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-kube-api-access-46zgk\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.778751 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.778718    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-socket\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.879854 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.879820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-data\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.879854 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.879859    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-credential-socket\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880100 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.879877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-podinfo\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880100 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.879905    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-envoy\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880100 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.879932    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istiod-ca-cert\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880100 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.879978    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-token\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880100 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.880037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-certs\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880100 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.880074    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zgk\" (UniqueName: \"kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-kube-api-access-46zgk\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880428 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.880113    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-socket\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880428 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.880251    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-data\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880428 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.880328    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-credential-socket\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.880427    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-socket\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.880552    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-certs\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.880740 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.880689    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istiod-ca-cert\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.882122 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.882092    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-envoy\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.882371 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.882355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-podinfo\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.893351 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.893322    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-token\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.893839 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.893816    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zgk\" (UniqueName: \"kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-kube-api-access-46zgk\") pod \"e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") " pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:06.908420 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.908401    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:57:06.908685 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.908667    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:57:06.921224 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.921173    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:57:06.978694 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:06.978665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:07.153622 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:07.153591    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 14:57:07.273120 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:07.273086    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"]
Sep 11 14:57:07.276254 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:57:07.276232    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fdd4bb_d339_479f_aaea_9bbeaa6276a4.slice/crio-7b935d4d4a5cc334fbfe1b077b53d608bd33bdea62ca99dab92b696a190bcfd2 WatchSource:0}: Error finding container 7b935d4d4a5cc334fbfe1b077b53d608bd33bdea62ca99dab92b696a190bcfd2: Status 404 returned error can't find the container with id 7b935d4d4a5cc334fbfe1b077b53d608bd33bdea62ca99dab92b696a190bcfd2
Sep 11 14:57:07.278566 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:07.278537    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:57:07.278620 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:07.278598    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:57:07.278661 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:07.278644    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:57:07.324886 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:07.324861    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:08.149490 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:08.149432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" event={"ID":"62fdd4bb-d339-479f-aaea-9bbeaa6276a4","Type":"ContainerStarted","Data":"80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6"}
Sep 11 14:57:08.149919 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:08.149504    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" event={"ID":"62fdd4bb-d339-479f-aaea-9bbeaa6276a4","Type":"ContainerStarted","Data":"7b935d4d4a5cc334fbfe1b077b53d608bd33bdea62ca99dab92b696a190bcfd2"}
Sep 11 14:57:08.175550 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:08.175505    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" podStartSLOduration=2.175487238 podStartE2EDuration="2.175487238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:57:08.17265676 +0000 UTC m=+1034.495285132" watchObservedRunningTime="2026-09-11 14:57:08.175487238 +0000 UTC m=+1034.498115607"
Sep 11 14:57:08.979051 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:08.979007    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:08.984243 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:08.984218    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:09.155184 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:09.155149    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:09.156243 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:09.156215    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:09.503915 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:09.503876    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:10.122627 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:10.122586    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:10.222586 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:10.222547    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:10.438010 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:10.437907    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:11.308832 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:11.308796    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:12.027861 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:12.027826    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:12.791354 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:12.791318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:13.705900 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:13.705864    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:13.801841 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:13.801809    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:14.210472 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:14.210404    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:14.357179 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:14.357143    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:14.636111 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:14.636076    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:17.129563 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:17.129505    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:17.971496 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:17.971435    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:24.170749 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:24.170708    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:24.277317 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:24.277284    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:24.379829 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:24.379782    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:24.476972 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:24.476937    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:25.275077 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:25.275033    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:25.362654 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:25.362611    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:25.463591 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:25.463560    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:27.041059 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:27.040996    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:57:27.106960 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:27.106922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:33.194948 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:33.194914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:33.267833 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:33.267748    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:33.917223 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:33.917177    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:35.978478 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:35.978195    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:36.820996 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:36.820952    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:37.543014 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:37.542971    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:38.407114 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:38.407086    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:39.181173 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:39.181132    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:40.014158 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:40.014109    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:40.326997 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:40.326904    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:41.661039 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:41.660953    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:41.767807 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:41.767771    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:41.921024 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:41.920914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:42.291908 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:42.291866    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:43.101148 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:43.101106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:45.343439 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:45.343400    2559 generic.go:356] "Generic (PLEG): container finished" podID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerID="80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6" exitCode=0
Sep 11 14:57:45.343886 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:45.343475    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" event={"ID":"62fdd4bb-d339-479f-aaea-9bbeaa6276a4","Type":"ContainerDied","Data":"80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6"}
Sep 11 14:57:45.343961 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:45.343942    2559 scope.go:117] "RemoveContainer" containerID="80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6"
Sep 11 14:57:45.345207 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:45.345180    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:57:45.345269 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:45.345250    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:57:45.351799 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:45.345421    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:57:46.350766 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:46.350732    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" event={"ID":"62fdd4bb-d339-479f-aaea-9bbeaa6276a4","Type":"ContainerStarted","Data":"b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e"}
Sep 11 14:57:46.789073 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:46.789037    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:46.849596 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:46.849560    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:46.979708 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:46.979661    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:46.979912 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:46.979776    2559 patch_prober.go:28] interesting pod/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 14:57:46.979912 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:46.979834    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 14:57:47.979243 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:47.979182    2559 patch_prober.go:28] interesting pod/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 14:57:47.979838 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:47.979279    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 14:57:48.712336 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:48.712296    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:48.979314 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:48.979271    2559 patch_prober.go:28] interesting pod/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 14:57:48.980631 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:48.980589    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 14:57:49.384357 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:49.384265    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:49.477516 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:49.477477    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:49.780974 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:49.780937    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:49.980153 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:49.980099    2559 patch_prober.go:28] interesting pod/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 14:57:49.980686 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:49.980188    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 14:57:50.528982 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:50.528943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:50.982950 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:50.982913    2559 patch_prober.go:28] interesting pod/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 14:57:50.983557 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:50.983524    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 14:57:51.416926 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:51.416833    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:52.017858 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:52.017820    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:52.018409 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:52.018192    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:52.019968 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:52.019942    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:57:52.227017 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:52.226985    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:52.260744 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:52.260705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:57:53.204173 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:53.204139    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:53.898248 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:53.898212    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"]
Sep 11 14:57:53.900951 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:53.900927    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:53.903685 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:53.903658    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-577852b1-serving-cert\""
Sep 11 14:57:53.914284 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:53.914256    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"]
Sep 11 14:57:53.929371 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:53.929336    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:53.961655 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:53.961618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-metrics-certs\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:53.961814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:53.961675    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-openshift-service-ca\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:53.961814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:53.961704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h58n\" (UniqueName: \"kubernetes.io/projected/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-kube-api-access-4h58n\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:53.961814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:53.961746    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-maas-api-tls\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:54.006374 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.006339    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:54.063111 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.063081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-maas-api-tls\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:54.063294 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.063191    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-metrics-certs\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:54.063294 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.063227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-openshift-service-ca\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:54.063294 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.063246    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h58n\" (UniqueName: \"kubernetes.io/projected/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-kube-api-access-4h58n\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:54.063966 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.063938    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-openshift-service-ca\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:54.065621 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.065584    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-metrics-certs\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:54.065715 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.065624    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-maas-api-tls\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:54.072437 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.072415    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h58n\" (UniqueName: \"kubernetes.io/projected/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-kube-api-access-4h58n\") pod \"maas-api-e2e-mt-577852b1-5f855548d9-vx55h\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:54.106814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.106780    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:54.216126 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.216032    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:54.426852 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.426820    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:54.494359 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.494332    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"]
Sep 11 14:57:54.495929 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:57:54.495902    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc7f96c_ccae_4a71_b77f_9ff80d3603aa.slice/crio-bc8faaa360abc93dc0541ca007260b1df1502ce53546407188dbd5a223150ac9 WatchSource:0}: Error finding container bc8faaa360abc93dc0541ca007260b1df1502ce53546407188dbd5a223150ac9: Status 404 returned error can't find the container with id bc8faaa360abc93dc0541ca007260b1df1502ce53546407188dbd5a223150ac9
Sep 11 14:57:54.497366 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.497348    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:57:54.531753 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:54.531721    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:57:55.406938 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:55.406903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h" event={"ID":"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa","Type":"ContainerStarted","Data":"4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012"}
Sep 11 14:57:55.407367 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:55.406953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h" event={"ID":"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa","Type":"ContainerStarted","Data":"bc8faaa360abc93dc0541ca007260b1df1502ce53546407188dbd5a223150ac9"}
Sep 11 14:57:55.407367 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:55.406990    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:57:55.425404 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:57:55.425360    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h" podStartSLOduration=2.037164795 podStartE2EDuration="2.425341051s" totalImagesPullingTime="388.176256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:57:55.424714098 +0000 UTC m=+1081.747342469" watchObservedRunningTime="2026-09-11 14:57:55.425341051 +0000 UTC m=+1081.747969422"
Sep 11 14:58:01.184977 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:01.184934    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:01.418784 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:01.418754    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:58:17.547339 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:17.547299    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:18.364298 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:18.364221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:19.386754 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:19.386703    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:19.611944 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:19.611909    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:20.149902 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.149865    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:20.276982 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.276949    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:20.375491 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.375432    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:20.599815 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.599779    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"]
Sep 11 14:58:20.603228 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.603205    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.606102 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.606081    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-87d086\"/\"kube-root-ca.crt\""
Sep 11 14:58:20.607388 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.607366    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-87d086\"/\"openshift-service-ca.crt\""
Sep 11 14:58:20.607537 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.607518    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-87d086\"/\"explicit-model-87d086-kserve-self-signed-certs\""
Sep 11 14:58:20.607593 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.607529    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-87d086\"/\"default-dockercfg-q4tkl\""
Sep 11 14:58:20.618189 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.618161    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"]
Sep 11 14:58:20.631437 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.631411    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:20.710319 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.710282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:20.731347 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.731318    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-model-cache\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.731498 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.731357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tls-certs\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.731498 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.731381    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tmp-dir\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.731498 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.731444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-home\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.731616 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.731533    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vjkl\" (UniqueName: \"kubernetes.io/projected/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-kube-api-access-9vjkl\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.731651 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.731621    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-dshm\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.832339 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.832297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-model-cache\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.832339 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.832346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tls-certs\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.832636 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.832379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tmp-dir\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.832636 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.832410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-home\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.832636 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.832497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vjkl\" (UniqueName: \"kubernetes.io/projected/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-kube-api-access-9vjkl\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.832636 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.832593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-dshm\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.832850 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.832769    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-model-cache\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.832850 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.832782    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tmp-dir\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.832960 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.832836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-home\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.834721 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.834699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-dshm\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.834981 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.834963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tls-certs\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.842914 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.842888    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vjkl\" (UniqueName: \"kubernetes.io/projected/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-kube-api-access-9vjkl\") pod \"explicit-model-87d086-kserve-fb446f66c-x2zfs\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") " pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:20.912858 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:20.912791    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:21.187793 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:21.187767    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"]
Sep 11 14:58:21.190125 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:58:21.190090    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8e3b0d6_d389_4362_aa5e_6f3495e7f6f2.slice/crio-be37db128f5ca349291b064359a5e744f557dc8747504f8cd24417b16533eff7 WatchSource:0}: Error finding container be37db128f5ca349291b064359a5e744f557dc8747504f8cd24417b16533eff7: Status 404 returned error can't find the container with id be37db128f5ca349291b064359a5e744f557dc8747504f8cd24417b16533eff7
Sep 11 14:58:21.195842 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:21.195813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:21.545786 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:21.545750    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs" event={"ID":"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2","Type":"ContainerStarted","Data":"373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058"}
Sep 11 14:58:21.545786 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:21.545790    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs" event={"ID":"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2","Type":"ContainerStarted","Data":"be37db128f5ca349291b064359a5e744f557dc8747504f8cd24417b16533eff7"}
Sep 11 14:58:21.583072 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:21.583023    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs" podStartSLOduration=1.5830030640000001 podStartE2EDuration="1.583003064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:58:21.580569735 +0000 UTC m=+1107.903198105" watchObservedRunningTime="2026-09-11 14:58:21.583003064 +0000 UTC m=+1107.905631436"
Sep 11 14:58:22.420019 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:22.419980    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:23.309406 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:23.309369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:27.249224 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:27.249183    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:27.349085 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:27.349051    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:27.921899 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:27.921863    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:58:28.696599 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:28.696562    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:29.406979 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:29.406944    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:30.104708 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:30.104666    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:30.910757 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:30.910725    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:30.913492 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:30.913475    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:30.913611 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:30.913513    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:30.925787 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:30.925765    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:31.583735 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:31.583702    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:31.608618 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:31.608593    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:58:32.439694 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:32.439658    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:33.128426 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:33.128382    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:38.189271 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.189238    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"]
Sep 11 14:58:38.189881 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.189482    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h" podUID="2cc7f96c-ccae-4a71-b77f-9ff80d3603aa" containerName="maas-api" containerID="cri-o://4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012" gracePeriod=30
Sep 11 14:58:38.223216 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.223183    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:38.305593 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.305559    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:38.405655 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.405620    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:38.443097 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.443034    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:58:38.503842 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.503795    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:38.581149 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.581113    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:38.621503 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.621444    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-metrics-certs\") pod \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") "
Sep 11 14:58:38.621692 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.621526    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-maas-api-tls\") pod \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") "
Sep 11 14:58:38.621692 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.621597    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h58n\" (UniqueName: \"kubernetes.io/projected/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-kube-api-access-4h58n\") pod \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") "
Sep 11 14:58:38.621815 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.621734    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-openshift-service-ca\") pod \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\" (UID: \"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa\") "
Sep 11 14:58:38.622115 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.622064    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2cc7f96c-ccae-4a71-b77f-9ff80d3603aa" (UID: "2cc7f96c-ccae-4a71-b77f-9ff80d3603aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:58:38.623777 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.623752    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-kube-api-access-4h58n" (OuterVolumeSpecName: "kube-api-access-4h58n") pod "2cc7f96c-ccae-4a71-b77f-9ff80d3603aa" (UID: "2cc7f96c-ccae-4a71-b77f-9ff80d3603aa"). InnerVolumeSpecName "kube-api-access-4h58n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:58:38.624083 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.624045    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2cc7f96c-ccae-4a71-b77f-9ff80d3603aa" (UID: "2cc7f96c-ccae-4a71-b77f-9ff80d3603aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:58:38.624183 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.624060    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2cc7f96c-ccae-4a71-b77f-9ff80d3603aa" (UID: "2cc7f96c-ccae-4a71-b77f-9ff80d3603aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:58:38.634318 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.634291    2559 generic.go:356] "Generic (PLEG): container finished" podID="2cc7f96c-ccae-4a71-b77f-9ff80d3603aa" containerID="4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012" exitCode=0
Sep 11 14:58:38.634440 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.634335    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"
Sep 11 14:58:38.634440 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.634368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h" event={"ID":"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa","Type":"ContainerDied","Data":"4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012"}
Sep 11 14:58:38.634440 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.634420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h" event={"ID":"2cc7f96c-ccae-4a71-b77f-9ff80d3603aa","Type":"ContainerDied","Data":"bc8faaa360abc93dc0541ca007260b1df1502ce53546407188dbd5a223150ac9"}
Sep 11 14:58:38.634627 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.634444    2559 scope.go:117] "RemoveContainer" containerID="4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012"
Sep 11 14:58:38.644392 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.644377    2559 scope.go:117] "RemoveContainer" containerID="4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012"
Sep 11 14:58:38.644713 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:58:38.644694    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012\": container with ID starting with 4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012 not found: ID does not exist" containerID="4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012"
Sep 11 14:58:38.644786 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.644723    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012"} err="failed to get container status \"4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012\": rpc error: code = NotFound desc = could not find container \"4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012\": container with ID starting with 4c9cbaace9eaad44ec157ff5a3f25d2389cca61d991c8fc1e86eabf9849cf012 not found: ID does not exist"
Sep 11 14:58:38.657095 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.657072    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"]
Sep 11 14:58:38.662993 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.662974    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-577852b1-5f855548d9-vx55h"]
Sep 11 14:58:38.722854 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.722826    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:58:38.722962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.722854    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h58n\" (UniqueName: \"kubernetes.io/projected/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-kube-api-access-4h58n\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:58:38.722962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.722868    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:58:38.722962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.722880    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:58:38.914805 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:38.914758    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:39.014988 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:39.014894    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:39.681971 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:39.681926    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:40.181476 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:40.181426    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cc7f96c-ccae-4a71-b77f-9ff80d3603aa" path="/var/lib/kubelet/pods/2cc7f96c-ccae-4a71-b77f-9ff80d3603aa/volumes"
Sep 11 14:58:40.419675 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:40.419631    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:41.306354 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:41.306312    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:42.108737 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:42.108698    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:42.896204 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:42.896164    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:43.818653 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:43.818621    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:44.578382 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:44.578343    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:45.715852 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:45.715805    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:45.908056 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:45.908019    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:46.008975 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:46.008940    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:46.312951 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:46.312857    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:46.503415 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:46.503372    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:46.612625 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:46.612536    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:46.710029 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:46.710001    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:46.822349 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:46.822318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:47.502687 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:47.502641    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:49.457563 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:49.457521    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"]
Sep 11 14:58:49.481431 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:49.481396    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:49.778618 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:49.778567    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:50.636015 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:50.635980    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:51.493614 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:51.493559    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:51.763608 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:51.763520    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"]
Sep 11 14:58:51.799199 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:51.799168    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:52.408367 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:52.408328    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:53.056612 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:53.056580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:53.819226 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:53.819195    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:53.929112 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:53.929079    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:54.049645 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:54.049610    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:54.143573 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:54.143480    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:54.301023 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:54.300985    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:58:54.799435 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:58:54.799395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:00.140854 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:00.140814    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:00.252028 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:00.251986    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:00.348927 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:00.348894    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:00.421829 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:00.421708    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:00.809918 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:00.809888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:00.910937 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:00.910901    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:01.369620 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:01.369580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:01.476257 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:01.476222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:01.584887 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:01.584852    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:02.229253 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:02.229218    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:59:02.366705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:02.366664    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:02.465557 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:02.465519    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:02.941086 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:02.941045    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:03.148057 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:03.148017    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:03.461076 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:03.461038    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:04.240800 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.240756    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:04.477484 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.477417    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs" podUID="a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" containerName="main" containerID="cri-o://373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058" gracePeriod=45
Sep 11 14:59:04.725505 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.725480    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:59:04.783179 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.783100    2559 generic.go:356] "Generic (PLEG): container finished" podID="a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" containerID="373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058" exitCode=0
Sep 11 14:59:04.783179 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.783139    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"
Sep 11 14:59:04.783179 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.783145    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs" event={"ID":"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2","Type":"ContainerDied","Data":"373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058"}
Sep 11 14:59:04.783437 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.783180    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs" event={"ID":"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2","Type":"ContainerDied","Data":"be37db128f5ca349291b064359a5e744f557dc8747504f8cd24417b16533eff7"}
Sep 11 14:59:04.783437 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.783199    2559 scope.go:117] "RemoveContainer" containerID="373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058"
Sep 11 14:59:04.791814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.791790    2559 scope.go:117] "RemoveContainer" containerID="373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058"
Sep 11 14:59:04.792108 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:04.792084    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058\": container with ID starting with 373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058 not found: ID does not exist" containerID="373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058"
Sep 11 14:59:04.792157 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.792126    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058"} err="failed to get container status \"373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058\": rpc error: code = NotFound desc = could not find container \"373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058\": container with ID starting with 373990e3862f7373000150246294dedc93580d87d44409c265465d8c3202a058 not found: ID does not exist"
Sep 11 14:59:04.886714 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.886684    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-home\") pod \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") "
Sep 11 14:59:04.886855 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.886729    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-dshm\") pod \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") "
Sep 11 14:59:04.886855 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.886793    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vjkl\" (UniqueName: \"kubernetes.io/projected/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-kube-api-access-9vjkl\") pod \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") "
Sep 11 14:59:04.886855 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.886816    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tmp-dir\") pod \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") "
Sep 11 14:59:04.886855 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.886833    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-model-cache\") pod \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") "
Sep 11 14:59:04.887081 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.886947    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-home" (OuterVolumeSpecName: "home") pod "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" (UID: "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:04.887081 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.886987    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tls-certs\") pod \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\" (UID: \"a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2\") "
Sep 11 14:59:04.887188 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.887082    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" (UID: "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:04.887188 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.887087    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" (UID: "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:04.887446 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.887400    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:04.887446 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.887418    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:04.887446 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.887426    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:04.889068 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.889041    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" (UID: "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:59:04.889161 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.889065    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-dshm" (OuterVolumeSpecName: "dshm") pod "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" (UID: "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:04.889161 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.889088    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-kube-api-access-9vjkl" (OuterVolumeSpecName: "kube-api-access-9vjkl") pod "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" (UID: "a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2"). InnerVolumeSpecName "kube-api-access-9vjkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:04.988863 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.988827    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:04.988863 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.988854    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:04.988863 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:04.988863    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vjkl\" (UniqueName: \"kubernetes.io/projected/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2-kube-api-access-9vjkl\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:05.109706 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:05.109672    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"]
Sep 11 14:59:05.111491 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:05.111469    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-87d086/explicit-model-87d086-kserve-fb446f66c-x2zfs"]
Sep 11 14:59:05.647378 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:05.647345    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:06.181007 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:06.180975    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" path="/var/lib/kubelet/pods/a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2/volumes"
Sep 11 14:59:06.522423 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:06.522394    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:06.615825 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:06.615793    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:06.783187 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:06.783075    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm" podUID="40f0f5ed-f848-431f-82c0-123bfcc68d58" containerName="main" containerID="cri-o://fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5" gracePeriod=45
Sep 11 14:59:07.027506 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.027483    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:59:07.108820 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.108739    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-tmp-dir\") pod \"40f0f5ed-f848-431f-82c0-123bfcc68d58\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") "
Sep 11 14:59:07.108820 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.108785    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f0f5ed-f848-431f-82c0-123bfcc68d58-tls-certs\") pod \"40f0f5ed-f848-431f-82c0-123bfcc68d58\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") "
Sep 11 14:59:07.108820 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.108820    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-model-cache\") pod \"40f0f5ed-f848-431f-82c0-123bfcc68d58\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") "
Sep 11 14:59:07.109094 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.108864    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-dshm\") pod \"40f0f5ed-f848-431f-82c0-123bfcc68d58\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") "
Sep 11 14:59:07.109094 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.108894    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-home\") pod \"40f0f5ed-f848-431f-82c0-123bfcc68d58\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") "
Sep 11 14:59:07.109094 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.108913    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlk5s\" (UniqueName: \"kubernetes.io/projected/40f0f5ed-f848-431f-82c0-123bfcc68d58-kube-api-access-hlk5s\") pod \"40f0f5ed-f848-431f-82c0-123bfcc68d58\" (UID: \"40f0f5ed-f848-431f-82c0-123bfcc68d58\") "
Sep 11 14:59:07.109094 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.109027    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40f0f5ed-f848-431f-82c0-123bfcc68d58" (UID: "40f0f5ed-f848-431f-82c0-123bfcc68d58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:07.109285 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.109098    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-home" (OuterVolumeSpecName: "home") pod "40f0f5ed-f848-431f-82c0-123bfcc68d58" (UID: "40f0f5ed-f848-431f-82c0-123bfcc68d58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:07.109285 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.109111    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-model-cache" (OuterVolumeSpecName: "model-cache") pod "40f0f5ed-f848-431f-82c0-123bfcc68d58" (UID: "40f0f5ed-f848-431f-82c0-123bfcc68d58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:07.109385 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.109284    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:07.109385 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.109300    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:07.109385 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.109308    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:07.111110 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.111082    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-dshm" (OuterVolumeSpecName: "dshm") pod "40f0f5ed-f848-431f-82c0-123bfcc68d58" (UID: "40f0f5ed-f848-431f-82c0-123bfcc68d58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:07.111110 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.111083    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f0f5ed-f848-431f-82c0-123bfcc68d58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40f0f5ed-f848-431f-82c0-123bfcc68d58" (UID: "40f0f5ed-f848-431f-82c0-123bfcc68d58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:59:07.111253 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.111110    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f0f5ed-f848-431f-82c0-123bfcc68d58-kube-api-access-hlk5s" (OuterVolumeSpecName: "kube-api-access-hlk5s") pod "40f0f5ed-f848-431f-82c0-123bfcc68d58" (UID: "40f0f5ed-f848-431f-82c0-123bfcc68d58"). InnerVolumeSpecName "kube-api-access-hlk5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:07.210730 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.210703    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f0f5ed-f848-431f-82c0-123bfcc68d58-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:07.210730 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.210726    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f0f5ed-f848-431f-82c0-123bfcc68d58-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:07.210730 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.210735    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlk5s\" (UniqueName: \"kubernetes.io/projected/40f0f5ed-f848-431f-82c0-123bfcc68d58-kube-api-access-hlk5s\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:07.481429 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.481396    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:07.802116 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.802028    2559 generic.go:356] "Generic (PLEG): container finished" podID="40f0f5ed-f848-431f-82c0-123bfcc68d58" containerID="fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5" exitCode=0
Sep 11 14:59:07.802116 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.802065    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"
Sep 11 14:59:07.802698 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.802116    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm" event={"ID":"40f0f5ed-f848-431f-82c0-123bfcc68d58","Type":"ContainerDied","Data":"fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5"}
Sep 11 14:59:07.802698 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.802159    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm" event={"ID":"40f0f5ed-f848-431f-82c0-123bfcc68d58","Type":"ContainerDied","Data":"b538f79beee2bb772ac0fb7b8fe4a9139a1368d5be0a6337251660a222e8ab92"}
Sep 11 14:59:07.802698 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.802178    2559 scope.go:117] "RemoveContainer" containerID="fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5"
Sep 11 14:59:07.811238 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.811220    2559 scope.go:117] "RemoveContainer" containerID="fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5"
Sep 11 14:59:07.811522 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:07.811503    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5\": container with ID starting with fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5 not found: ID does not exist" containerID="fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5"
Sep 11 14:59:07.811588 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.811531    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5"} err="failed to get container status \"fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5\": rpc error: code = NotFound desc = could not find container \"fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5\": container with ID starting with fc564294025b8729ff5422dbdf8e80b44178101cdaaf65d22df89a5b28d2efa5 not found: ID does not exist"
Sep 11 14:59:07.838990 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.838963    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"]
Sep 11 14:59:07.844005 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:07.843984    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-362f7a/test-model-362f7a-kserve-76884c6587-w7cbm"]
Sep 11 14:59:08.179819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:08.179740    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40f0f5ed-f848-431f-82c0-123bfcc68d58" path="/var/lib/kubelet/pods/40f0f5ed-f848-431f-82c0-123bfcc68d58/volumes"
Sep 11 14:59:08.560692 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:08.560663    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"]
Sep 11 14:59:08.560881 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:08.560860    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy" containerID="cri-o://b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e" gracePeriod=30
Sep 11 14:59:08.650101 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:08.650064    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:08.750230 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:08.750194    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:09.025131 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.025086    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:09.234964 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.234926    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:09.323199 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.323117    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"]
Sep 11 14:59:09.323678 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.323660    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc7f96c-ccae-4a71-b77f-9ff80d3603aa" containerName="maas-api"
Sep 11 14:59:09.323678 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.323679    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc7f96c-ccae-4a71-b77f-9ff80d3603aa" containerName="maas-api"
Sep 11 14:59:09.323819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.323703    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f0f5ed-f848-431f-82c0-123bfcc68d58" containerName="main"
Sep 11 14:59:09.323819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.323709    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f0f5ed-f848-431f-82c0-123bfcc68d58" containerName="main"
Sep 11 14:59:09.323819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.323717    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" containerName="main"
Sep 11 14:59:09.323819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.323722    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" containerName="main"
Sep 11 14:59:09.323819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.323802    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc7f96c-ccae-4a71-b77f-9ff80d3603aa" containerName="maas-api"
Sep 11 14:59:09.323819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.323813    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8e3b0d6-d389-4362-aa5e-6f3495e7f6f2" containerName="main"
Sep 11 14:59:09.323819 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.323819    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="40f0f5ed-f848-431f-82c0-123bfcc68d58" containerName="main"
Sep 11 14:59:09.328603 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.328578    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.331501 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.331476    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-cd2cc1-openshift-default-dockercfg-q9hx2\""
Sep 11 14:59:09.341129 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.341099    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"]
Sep 11 14:59:09.354110 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.354078    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:09.432562 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.432527    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-credential-socket\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.432562 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.432564    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-podinfo\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.432809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.432585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-envoy\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.432809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.432677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-certs\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.432809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.432731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-data\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.432809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.432758    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-token\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.432809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.432789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-socket\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.433044 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.432851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlbt\" (UniqueName: \"kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-kube-api-access-5dlbt\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.433044 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.432922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9644e5d-0b72-475b-be30-72a4674e2d07-istiod-ca-cert\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.451903 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.451867    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:09.534129 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.534098    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-credential-socket\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.534129 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.534135    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-podinfo\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.534372 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.534157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-envoy\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.534372 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.534187    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-certs\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.534372 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.534224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-data\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.534372 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.534251    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-token\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.534604 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.534407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-socket\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.534604 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.534505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlbt\" (UniqueName: \"kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-kube-api-access-5dlbt\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.534604 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.534542    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-credential-socket\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.534604 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.534563    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9644e5d-0b72-475b-be30-72a4674e2d07-istiod-ca-cert\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.535113 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.535083    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-data\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.535229 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.535141    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-socket\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.535229 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.535152    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9644e5d-0b72-475b-be30-72a4674e2d07-istiod-ca-cert\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.535379 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.535361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-certs\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.536550 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.536528    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-envoy\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.537002 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.536981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-podinfo\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.544883 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.544859    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-token\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.545005 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.544903    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlbt\" (UniqueName: \"kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-kube-api-access-5dlbt\") pod \"e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") " pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.641352 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.641278    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:09.930029 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.930005    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"]
Sep 11 14:59:09.931942 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:59:09.931910    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9644e5d_0b72_475b_be30_72a4674e2d07.slice/crio-78ba9e0dc462c2cc6dae9afa3bbaf25ef12f8e5884f33c0ed537e1c2d1d0f932 WatchSource:0}: Error finding container 78ba9e0dc462c2cc6dae9afa3bbaf25ef12f8e5884f33c0ed537e1c2d1d0f932: Status 404 returned error can't find the container with id 78ba9e0dc462c2cc6dae9afa3bbaf25ef12f8e5884f33c0ed537e1c2d1d0f932
Sep 11 14:59:09.934242 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.934207    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:59:09.934350 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.934297    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:59:09.934350 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.934341    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:59:09.999831 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:09.999804    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"]
Sep 11 14:59:10.011598 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.011570    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.014594 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.014573    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c4e39b-serving-cert\""
Sep 11 14:59:10.017736 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.017707    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"]
Sep 11 14:59:10.038344 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.038298    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:10.128501 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.128470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:10.139724 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.139693    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfc0283-a6ef-4658-9584-ad55c27104ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.139862 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.139739    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.139862 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.139849    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.139948 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.139880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f4dr\" (UniqueName: \"kubernetes.io/projected/7dfc0283-a6ef-4658-9584-ad55c27104ac-kube-api-access-2f4dr\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.240564 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.240532    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.240720 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.240579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f4dr\" (UniqueName: \"kubernetes.io/projected/7dfc0283-a6ef-4658-9584-ad55c27104ac-kube-api-access-2f4dr\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.240720 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.240622    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfc0283-a6ef-4658-9584-ad55c27104ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.240720 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.240650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.240886 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:10.240830    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-serving-cert: secret "maas-api-e2e-shared-a-w2-c4e39b-serving-cert" not found
Sep 11 14:59:10.240946 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:10.240888    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-maas-api-tls podName:7dfc0283-a6ef-4658-9584-ad55c27104ac nodeName:}" failed. No retries permitted until 2026-09-11 14:59:10.740868477 +0000 UTC m=+1157.063496830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-maas-api-tls") pod "maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k" (UID: "7dfc0283-a6ef-4658-9584-ad55c27104ac") : secret "maas-api-e2e-shared-a-w2-c4e39b-serving-cert" not found
Sep 11 14:59:10.241309 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.241284    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfc0283-a6ef-4658-9584-ad55c27104ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.242995 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.242976    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.251485 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.251441    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f4dr\" (UniqueName: \"kubernetes.io/projected/7dfc0283-a6ef-4658-9584-ad55c27104ac-kube-api-access-2f4dr\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.403084 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.403049    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:10.668427 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.668341    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:10.747162 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.747127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.749634 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.749610    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:10.768776 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.768744    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:10.821234 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.821196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9" event={"ID":"c9644e5d-0b72-475b-be30-72a4674e2d07","Type":"ContainerStarted","Data":"3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a"}
Sep 11 14:59:10.821397 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.821271    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9" event={"ID":"c9644e5d-0b72-475b-be30-72a4674e2d07","Type":"ContainerStarted","Data":"78ba9e0dc462c2cc6dae9afa3bbaf25ef12f8e5884f33c0ed537e1c2d1d0f932"}
Sep 11 14:59:10.857746 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.857698    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9" podStartSLOduration=1.857680821 podStartE2EDuration="1.857680821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:10.855550186 +0000 UTC m=+1157.178178555" watchObservedRunningTime="2026-09-11 14:59:10.857680821 +0000 UTC m=+1157.180309248"
Sep 11 14:59:10.925666 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:10.925597    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:11.159814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.159772    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:11.212667 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.212646    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"]
Sep 11 14:59:11.214649 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:59:11.214621    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfc0283_a6ef_4658_9584_ad55c27104ac.slice/crio-aa64092859c4f61d35debd00d02456c025dece4f0b6c6acf87d1344a1a681efc WatchSource:0}: Error finding container aa64092859c4f61d35debd00d02456c025dece4f0b6c6acf87d1344a1a681efc: Status 404 returned error can't find the container with id aa64092859c4f61d35debd00d02456c025dece4f0b6c6acf87d1344a1a681efc
Sep 11 14:59:11.248221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.248189    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:11.365928 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.365883    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:11.642505 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.642469    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:11.648298 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.648266    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:11.827949 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.827911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k" event={"ID":"7dfc0283-a6ef-4658-9584-ad55c27104ac","Type":"ContainerStarted","Data":"a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad"}
Sep 11 14:59:11.828108 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.827963    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k" event={"ID":"7dfc0283-a6ef-4658-9584-ad55c27104ac","Type":"ContainerStarted","Data":"aa64092859c4f61d35debd00d02456c025dece4f0b6c6acf87d1344a1a681efc"}
Sep 11 14:59:11.828108 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.828062    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:11.829091 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.829072    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 14:59:11.852743 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.852699    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k" podStartSLOduration=2.342766991 podStartE2EDuration="2.852684117s" totalImagesPullingTime="509.917126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:11.850754725 +0000 UTC m=+1158.173383097" watchObservedRunningTime="2026-09-11 14:59:11.852684117 +0000 UTC m=+1158.175312488"
Sep 11 14:59:11.943478 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:11.943368    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:12.788784 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:12.788753    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:12.833784 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:12.833756    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:13.817191 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.817167    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:59:13.851346 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.851261    2559 generic.go:356] "Generic (PLEG): container finished" podID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerID="b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e" exitCode=0
Sep 11 14:59:13.851346 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.851332    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"
Sep 11 14:59:13.851587 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.851345    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" event={"ID":"62fdd4bb-d339-479f-aaea-9bbeaa6276a4","Type":"ContainerDied","Data":"b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e"}
Sep 11 14:59:13.851587 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.851441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm" event={"ID":"62fdd4bb-d339-479f-aaea-9bbeaa6276a4","Type":"ContainerDied","Data":"7b935d4d4a5cc334fbfe1b077b53d608bd33bdea62ca99dab92b696a190bcfd2"}
Sep 11 14:59:13.851587 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.851490    2559 scope.go:117] "RemoveContainer" containerID="b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e"
Sep 11 14:59:13.860330 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.860305    2559 scope.go:117] "RemoveContainer" containerID="80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6"
Sep 11 14:59:13.868366 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.868339    2559 scope.go:117] "RemoveContainer" containerID="b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e"
Sep 11 14:59:13.868648 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:13.868628    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e\": container with ID starting with b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e not found: ID does not exist" containerID="b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e"
Sep 11 14:59:13.868734 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.868656    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e"} err="failed to get container status \"b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e\": rpc error: code = NotFound desc = could not find container \"b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e\": container with ID starting with b4b9cbe3f63746334b3720496d9cf21ce001b0d36eaeb8ee9a144654533aa99e not found: ID does not exist"
Sep 11 14:59:13.868734 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.868673    2559 scope.go:117] "RemoveContainer" containerID="80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6"
Sep 11 14:59:13.868942 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:13.868924    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6\": container with ID starting with 80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6 not found: ID does not exist" containerID="80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6"
Sep 11 14:59:13.868984 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.868948    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6"} err="failed to get container status \"80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6\": rpc error: code = NotFound desc = could not find container \"80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6\": container with ID starting with 80dd8f92cdda1b0705c31291d211b8826eca683d7e2dd66cddbd988909c076d6 not found: ID does not exist"
Sep 11 14:59:13.980050 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980021    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-data\") pod \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") "
Sep 11 14:59:13.980050 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980053    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istiod-ca-cert\") pod \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") "
Sep 11 14:59:13.980270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980104    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-credential-socket\") pod \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") "
Sep 11 14:59:13.980270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980121    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46zgk\" (UniqueName: \"kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-kube-api-access-46zgk\") pod \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") "
Sep 11 14:59:13.980270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980154    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-certs\") pod \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") "
Sep 11 14:59:13.980270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980202    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-socket\") pod \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") "
Sep 11 14:59:13.980270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980223    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-envoy\") pod \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") "
Sep 11 14:59:13.980270 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980258    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-token\") pod \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") "
Sep 11 14:59:13.980662 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980324    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-podinfo\") pod \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\" (UID: \"62fdd4bb-d339-479f-aaea-9bbeaa6276a4\") "
Sep 11 14:59:13.980662 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980322    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "62fdd4bb-d339-479f-aaea-9bbeaa6276a4" (UID: "62fdd4bb-d339-479f-aaea-9bbeaa6276a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:13.980662 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980352    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62fdd4bb-d339-479f-aaea-9bbeaa6276a4" (UID: "62fdd4bb-d339-479f-aaea-9bbeaa6276a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:13.980662 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980577    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62fdd4bb-d339-479f-aaea-9bbeaa6276a4" (UID: "62fdd4bb-d339-479f-aaea-9bbeaa6276a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:13.980662 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980585    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62fdd4bb-d339-479f-aaea-9bbeaa6276a4" (UID: "62fdd4bb-d339-479f-aaea-9bbeaa6276a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:59:13.980946 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980774    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-data\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:13.980946 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980793    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istiod-ca-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:13.980946 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980807    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-credential-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:13.980946 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.980820    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:13.981141 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.981112    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62fdd4bb-d339-479f-aaea-9bbeaa6276a4" (UID: "62fdd4bb-d339-479f-aaea-9bbeaa6276a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:13.982757 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.982736    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-kube-api-access-46zgk" (OuterVolumeSpecName: "kube-api-access-46zgk") pod "62fdd4bb-d339-479f-aaea-9bbeaa6276a4" (UID: "62fdd4bb-d339-479f-aaea-9bbeaa6276a4"). InnerVolumeSpecName "kube-api-access-46zgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:13.982942 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.982919    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62fdd4bb-d339-479f-aaea-9bbeaa6276a4" (UID: "62fdd4bb-d339-479f-aaea-9bbeaa6276a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:59:13.982990 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.982976    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "62fdd4bb-d339-479f-aaea-9bbeaa6276a4" (UID: "62fdd4bb-d339-479f-aaea-9bbeaa6276a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:13.983974 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:13.983950    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62fdd4bb-d339-479f-aaea-9bbeaa6276a4" (UID: "62fdd4bb-d339-479f-aaea-9bbeaa6276a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:14.082016 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:14.081978    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-workload-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:14.082016 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:14.082009    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-envoy\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:14.082016 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:14.082021    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-token\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:14.082237 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:14.082032    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-istio-podinfo\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:14.082237 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:14.082044    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46zgk\" (UniqueName: \"kubernetes.io/projected/62fdd4bb-d339-479f-aaea-9bbeaa6276a4-kube-api-access-46zgk\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:14.183388 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:14.183363    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"]
Sep 11 14:59:14.183543 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:14.183395    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-87d086-openshift-default-7ff69c69b-4dwsm"]
Sep 11 14:59:16.180180 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:16.180143    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" path="/var/lib/kubelet/pods/62fdd4bb-d339-479f-aaea-9bbeaa6276a4/volumes"
Sep 11 14:59:18.857223 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:18.857182    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 14:59:23.023678 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:23.023637    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:23.885962 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:23.885927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:24.607593 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:24.607561    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:25.491958 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:25.491920    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:26.188010 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.187972    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:26.282919 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.282883    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:26.558245 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.558206    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:26.881841 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.881760    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"]
Sep 11 14:59:26.882179 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.882166    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy"
Sep 11 14:59:26.882222 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.882181    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy"
Sep 11 14:59:26.882222 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.882199    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy"
Sep 11 14:59:26.882222 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.882204    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy"
Sep 11 14:59:26.882322 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.882278    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy"
Sep 11 14:59:26.882322 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.882286    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="62fdd4bb-d339-479f-aaea-9bbeaa6276a4" containerName="istio-proxy"
Sep 11 14:59:26.885664 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.885636    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:26.888301 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.888277    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c0e28e-openshift-default-dockercfg-w2cgs\""
Sep 11 14:59:26.900844 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.900812    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"]
Sep 11 14:59:26.917101 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:26.917062    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:27.010638 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.010598    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwn4z\" (UniqueName: \"kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-kube-api-access-zwn4z\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.010809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.010660    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-token\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.010809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.010706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-credential-socket\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.010809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.010745    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-data\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.010809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.010772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-envoy\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.010809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.010804    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81c2c16e-25b6-4584-9190-1fd68da51f25-istiod-ca-cert\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.010985 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.010835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-certs\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.010985 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.010857    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-socket\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.010985 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.010926    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-podinfo\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.070574 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.070535    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:27.112137 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.112102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwn4z\" (UniqueName: \"kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-kube-api-access-zwn4z\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.112314 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.112149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-token\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.112314 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.112260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-credential-socket\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.112392 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.112327    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-data\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.112392 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.112349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-envoy\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.112392 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.112385    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81c2c16e-25b6-4584-9190-1fd68da51f25-istiod-ca-cert\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.112577 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.112433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-certs\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.112577 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.112500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-socket\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.112577 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.112547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-podinfo\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.113017 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.112990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-credential-socket\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.113193 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.113163    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-certs\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.113282 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.113190    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-data\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.113339 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.113294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-socket\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.113397 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.113336    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81c2c16e-25b6-4584-9190-1fd68da51f25-istiod-ca-cert\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.115536 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.115514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-envoy\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.115698 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.115676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-podinfo\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.121069 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.121044    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwn4z\" (UniqueName: \"kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-kube-api-access-zwn4z\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.121328 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.121309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-token\") pod \"e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") " pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.196166 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.196084    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:27.215509 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.215476    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:27.478313 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.478286    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"]
Sep 11 14:59:27.480197 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:59:27.480157    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c2c16e_25b6_4584_9190_1fd68da51f25.slice/crio-c005e5a0c626d630c248f036ba130eba382ca61ad6b97f0ba4c5f535d8fcb35e WatchSource:0}: Error finding container c005e5a0c626d630c248f036ba130eba382ca61ad6b97f0ba4c5f535d8fcb35e: Status 404 returned error can't find the container with id c005e5a0c626d630c248f036ba130eba382ca61ad6b97f0ba4c5f535d8fcb35e
Sep 11 14:59:27.482855 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.482810    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:59:27.482945 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.482906    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:59:27.482990 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.482953    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:59:27.929285 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.929178    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" event={"ID":"81c2c16e-25b6-4584-9190-1fd68da51f25","Type":"ContainerStarted","Data":"b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf"}
Sep 11 14:59:27.929285 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.929242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" event={"ID":"81c2c16e-25b6-4584-9190-1fd68da51f25","Type":"ContainerStarted","Data":"c005e5a0c626d630c248f036ba130eba382ca61ad6b97f0ba4c5f535d8fcb35e"}
Sep 11 14:59:27.952590 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:27.952539    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" podStartSLOduration=1.952521462 podStartE2EDuration="1.952521462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:27.950564549 +0000 UTC m=+1174.273192919" watchObservedRunningTime="2026-09-11 14:59:27.952521462 +0000 UTC m=+1174.275149831"
Sep 11 14:59:28.069628 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:28.069589    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:28.196587 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:28.196503    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:28.201809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:28.201782    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:28.764084 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:28.764046    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:28.934352 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:28.934316    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:28.935259 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:28.935239    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:29.028248 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:29.028166    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 14:59:29.631014 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:29.630983    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:30.083631 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:30.083596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:30.185031 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:30.184996    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:30.625005 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:30.624969    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:31.529294 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:31.529260    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:31.629180 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:31.629146    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:31.827015 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:31.826924    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:32.083945 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:32.083855    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:32.186290 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:32.186254    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:32.566636 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:32.566596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:33.461019 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:33.460935    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:34.177378 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:34.177323    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:34.283769 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:34.283730    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:34.377375 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:34.377336    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:34.482022 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:34.481939    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:34.588403 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:34.588366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:34.902519 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:34.902387    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:35.544667 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:35.544632    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:35.646724 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:35.646687    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:36.351211 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:36.351174    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:37.415521 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:37.415483    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:38.124932 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:38.124898    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:39.005720 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:39.005686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:39.712060 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:39.712018    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:41.556987 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:41.556940    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:41.658546 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:41.658500    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:41.852702 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:41.852604    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:42.520383 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:42.520346    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:43.506787 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:43.506748    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:44.336392 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:44.336357    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:44.705042 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:44.704949    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:45.239666 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:45.239626    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:45.529488 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:45.529383    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7671-db9ec5"]
Sep 11 14:59:45.534152 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:45.534128    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-db9ec5"
Sep 11 14:59:45.536806 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:45.536789    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5lj77\""
Sep 11 14:59:45.541209 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:45.541185    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-db9ec5"]
Sep 11 14:59:45.711753 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:45.711717    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plzhz\" (UniqueName: \"kubernetes.io/projected/efa1ce73-a37c-47c2-a63c-3b5978be0c4f-kube-api-access-plzhz\") pod \"test-curl-7671-db9ec5\" (UID: \"efa1ce73-a37c-47c2-a63c-3b5978be0c4f\") " pod="opendatahub/test-curl-7671-db9ec5"
Sep 11 14:59:45.813209 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:45.813115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plzhz\" (UniqueName: \"kubernetes.io/projected/efa1ce73-a37c-47c2-a63c-3b5978be0c4f-kube-api-access-plzhz\") pod \"test-curl-7671-db9ec5\" (UID: \"efa1ce73-a37c-47c2-a63c-3b5978be0c4f\") " pod="opendatahub/test-curl-7671-db9ec5"
Sep 11 14:59:45.821339 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:45.821312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plzhz\" (UniqueName: \"kubernetes.io/projected/efa1ce73-a37c-47c2-a63c-3b5978be0c4f-kube-api-access-plzhz\") pod \"test-curl-7671-db9ec5\" (UID: \"efa1ce73-a37c-47c2-a63c-3b5978be0c4f\") " pod="opendatahub/test-curl-7671-db9ec5"
Sep 11 14:59:45.846032 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:45.846000    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-db9ec5"
Sep 11 14:59:46.042342 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:46.042306    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:46.137047 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:46.137019    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-db9ec5"]
Sep 11 14:59:46.139382 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:59:46.139352    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefa1ce73_a37c_47c2_a63c_3b5978be0c4f.slice/crio-4f1bdc9e785ab5a30beff686d0384d63f0646dd71a5574299cc6f62e26a9a660 WatchSource:0}: Error finding container 4f1bdc9e785ab5a30beff686d0384d63f0646dd71a5574299cc6f62e26a9a660: Status 404 returned error can't find the container with id 4f1bdc9e785ab5a30beff686d0384d63f0646dd71a5574299cc6f62e26a9a660
Sep 11 14:59:47.025014 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.024975    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:47.029267 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.029238    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-db9ec5" event={"ID":"efa1ce73-a37c-47c2-a63c-3b5978be0c4f","Type":"ContainerStarted","Data":"2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306"}
Sep 11 14:59:47.029384 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.029276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-db9ec5" event={"ID":"efa1ce73-a37c-47c2-a63c-3b5978be0c4f","Type":"ContainerStarted","Data":"4f1bdc9e785ab5a30beff686d0384d63f0646dd71a5574299cc6f62e26a9a660"}
Sep 11 14:59:47.045191 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.045144    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7671-db9ec5" podStartSLOduration=2.045130482 podStartE2EDuration="2.045130482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:47.043897682 +0000 UTC m=+1193.366526053" watchObservedRunningTime="2026-09-11 14:59:47.045130482 +0000 UTC m=+1193.367758861"
Sep 11 14:59:47.749950 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.749864    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7671-db9ec5"]
Sep 11 14:59:47.755116 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.755088    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7671-db9ec5"]
Sep 11 14:59:47.759481 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.756841    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:47.883501 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.877143    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7671-63b94f"]
Sep 11 14:59:47.884221 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.884189    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa1ce73-a37c-47c2-a63c-3b5978be0c4f" containerName="test-curl-7671-db9ec5"
Sep 11 14:59:47.884391 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.884377    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa1ce73-a37c-47c2-a63c-3b5978be0c4f" containerName="test-curl-7671-db9ec5"
Sep 11 14:59:47.884648 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.884630    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa1ce73-a37c-47c2-a63c-3b5978be0c4f" containerName="test-curl-7671-db9ec5"
Sep 11 14:59:47.888661 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.888630    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-63b94f"]
Sep 11 14:59:47.888901 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.888856    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-63b94f"
Sep 11 14:59:47.935029 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:47.934999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ch7\" (UniqueName: \"kubernetes.io/projected/9e88467e-02a2-4458-b85d-69f46bf776d2-kube-api-access-87ch7\") pod \"test-curl-7671-63b94f\" (UID: \"9e88467e-02a2-4458-b85d-69f46bf776d2\") " pod="opendatahub/test-curl-7671-63b94f"
Sep 11 14:59:48.034654 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:48.034574    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7671-db9ec5" container="test-curl-7671-db9ec5"
Sep 11 14:59:48.035910 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:48.035889    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ch7\" (UniqueName: \"kubernetes.io/projected/9e88467e-02a2-4458-b85d-69f46bf776d2-kube-api-access-87ch7\") pod \"test-curl-7671-63b94f\" (UID: \"9e88467e-02a2-4458-b85d-69f46bf776d2\") " pod="opendatahub/test-curl-7671-63b94f"
Sep 11 14:59:48.044316 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:48.044283    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ch7\" (UniqueName: \"kubernetes.io/projected/9e88467e-02a2-4458-b85d-69f46bf776d2-kube-api-access-87ch7\") pod \"test-curl-7671-63b94f\" (UID: \"9e88467e-02a2-4458-b85d-69f46bf776d2\") " pod="opendatahub/test-curl-7671-63b94f"
Sep 11 14:59:48.200200 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:48.200167    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-63b94f"
Sep 11 14:59:48.472249 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:48.472217    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-63b94f"]
Sep 11 14:59:48.474771 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:59:48.474745    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e88467e_02a2_4458_b85d_69f46bf776d2.slice/crio-05c65194cbaabe99cd86eaba3016aa9d420419ef41fef1195feb22f23902ece2 WatchSource:0}: Error finding container 05c65194cbaabe99cd86eaba3016aa9d420419ef41fef1195feb22f23902ece2: Status 404 returned error can't find the container with id 05c65194cbaabe99cd86eaba3016aa9d420419ef41fef1195feb22f23902ece2
Sep 11 14:59:48.609555 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:48.609512    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:49.041618 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.041583    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-63b94f" event={"ID":"9e88467e-02a2-4458-b85d-69f46bf776d2","Type":"ContainerStarted","Data":"24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690"}
Sep 11 14:59:49.042075 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.041627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-63b94f" event={"ID":"9e88467e-02a2-4458-b85d-69f46bf776d2","Type":"ContainerStarted","Data":"05c65194cbaabe99cd86eaba3016aa9d420419ef41fef1195feb22f23902ece2"}
Sep 11 14:59:49.042075 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.041706    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7671-db9ec5" podUID="efa1ce73-a37c-47c2-a63c-3b5978be0c4f" containerName="test-curl-7671-db9ec5" containerID="cri-o://2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306" gracePeriod=2
Sep 11 14:59:49.056921 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.056865    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7671-63b94f" podStartSLOduration=2.0568458 podStartE2EDuration="2.0568458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:49.056257933 +0000 UTC m=+1195.378886303" watchObservedRunningTime="2026-09-11 14:59:49.0568458 +0000 UTC m=+1195.379474174"
Sep 11 14:59:49.343472 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.343377    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:49.722732 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.722696    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7671-63b94f"]
Sep 11 14:59:49.729332 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.729300    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7671-63b94f"]
Sep 11 14:59:49.848000 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.847963    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7671-f506bf"]
Sep 11 14:59:49.848653 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.848632    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e88467e-02a2-4458-b85d-69f46bf776d2" containerName="test-curl-7671-63b94f"
Sep 11 14:59:49.848700 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.848656    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e88467e-02a2-4458-b85d-69f46bf776d2" containerName="test-curl-7671-63b94f"
Sep 11 14:59:49.848799 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.848784    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e88467e-02a2-4458-b85d-69f46bf776d2" containerName="test-curl-7671-63b94f"
Sep 11 14:59:49.852448 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.852429    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-f506bf"
Sep 11 14:59:49.857504 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.857411    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-f506bf"]
Sep 11 14:59:49.956212 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:49.956168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpmqh\" (UniqueName: \"kubernetes.io/projected/a2a4759b-8c66-418f-b071-147ed25ca881-kube-api-access-rpmqh\") pod \"test-curl-7671-f506bf\" (UID: \"a2a4759b-8c66-418f-b071-147ed25ca881\") " pod="opendatahub/test-curl-7671-f506bf"
Sep 11 14:59:50.047225 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:50.047133    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7671-63b94f" container="test-curl-7671-63b94f"
Sep 11 14:59:50.057435 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:50.057408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmqh\" (UniqueName: \"kubernetes.io/projected/a2a4759b-8c66-418f-b071-147ed25ca881-kube-api-access-rpmqh\") pod \"test-curl-7671-f506bf\" (UID: \"a2a4759b-8c66-418f-b071-147ed25ca881\") " pod="opendatahub/test-curl-7671-f506bf"
Sep 11 14:59:50.066107 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:50.066073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmqh\" (UniqueName: \"kubernetes.io/projected/a2a4759b-8c66-418f-b071-147ed25ca881-kube-api-access-rpmqh\") pod \"test-curl-7671-f506bf\" (UID: \"a2a4759b-8c66-418f-b071-147ed25ca881\") " pod="opendatahub/test-curl-7671-f506bf"
Sep 11 14:59:50.165958 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:50.165922    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-f506bf"
Sep 11 14:59:50.437372 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:50.437347    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-f506bf"]
Sep 11 14:59:50.440255 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:59:50.440222    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2a4759b_8c66_418f_b071_147ed25ca881.slice/crio-99ffee4309fe058111f8aa9837bf5c972f0a4f59e1391bb80a4f0f5d11308a03 WatchSource:0}: Error finding container 99ffee4309fe058111f8aa9837bf5c972f0a4f59e1391bb80a4f0f5d11308a03: Status 404 returned error can't find the container with id 99ffee4309fe058111f8aa9837bf5c972f0a4f59e1391bb80a4f0f5d11308a03
Sep 11 14:59:50.861079 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:50.861042    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:51.055848 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.055805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-f506bf" event={"ID":"a2a4759b-8c66-418f-b071-147ed25ca881","Type":"ContainerStarted","Data":"1278120915b167ee48a9238c07c85af69a5a67060546001fd34fbcffb0997363"}
Sep 11 14:59:51.056208 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.055865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-f506bf" event={"ID":"a2a4759b-8c66-418f-b071-147ed25ca881","Type":"ContainerStarted","Data":"99ffee4309fe058111f8aa9837bf5c972f0a4f59e1391bb80a4f0f5d11308a03"}
Sep 11 14:59:51.056208 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.055975    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7671-63b94f" podUID="9e88467e-02a2-4458-b85d-69f46bf776d2" containerName="test-curl-7671-63b94f" containerID="cri-o://24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690" gracePeriod=2
Sep 11 14:59:51.071641 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.071590    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7671-f506bf" podStartSLOduration=2.071571887 podStartE2EDuration="2.071571887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:51.071346569 +0000 UTC m=+1197.393974936" watchObservedRunningTime="2026-09-11 14:59:51.071571887 +0000 UTC m=+1197.394200259"
Sep 11 14:59:51.212836 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.212807    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-db9ec5"
Sep 11 14:59:51.269835 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.269804    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plzhz\" (UniqueName: \"kubernetes.io/projected/efa1ce73-a37c-47c2-a63c-3b5978be0c4f-kube-api-access-plzhz\") pod \"efa1ce73-a37c-47c2-a63c-3b5978be0c4f\" (UID: \"efa1ce73-a37c-47c2-a63c-3b5978be0c4f\") "
Sep 11 14:59:51.271794 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.271768    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa1ce73-a37c-47c2-a63c-3b5978be0c4f-kube-api-access-plzhz" (OuterVolumeSpecName: "kube-api-access-plzhz") pod "efa1ce73-a37c-47c2-a63c-3b5978be0c4f" (UID: "efa1ce73-a37c-47c2-a63c-3b5978be0c4f"). InnerVolumeSpecName "kube-api-access-plzhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:51.371218 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.371182    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plzhz\" (UniqueName: \"kubernetes.io/projected/efa1ce73-a37c-47c2-a63c-3b5978be0c4f-kube-api-access-plzhz\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:51.669781 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.669744    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7671-f506bf"]
Sep 11 14:59:51.677520 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.677494    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7671-f506bf"]
Sep 11 14:59:51.787294 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.787252    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7671-f1b1b7"]
Sep 11 14:59:51.787771 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.787750    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2a4759b-8c66-418f-b071-147ed25ca881" containerName="test-curl-7671-f506bf"
Sep 11 14:59:51.787883 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.787772    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2a4759b-8c66-418f-b071-147ed25ca881" containerName="test-curl-7671-f506bf"
Sep 11 14:59:51.787883 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.787873    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2a4759b-8c66-418f-b071-147ed25ca881" containerName="test-curl-7671-f506bf"
Sep 11 14:59:51.790913 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.790892    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-f1b1b7"
Sep 11 14:59:51.795644 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.795618    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-f1b1b7"]
Sep 11 14:59:51.877167 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.877138    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqlx2\" (UniqueName: \"kubernetes.io/projected/12e9064a-baeb-44e6-b8cd-4dc046e75efe-kube-api-access-xqlx2\") pod \"test-curl-7671-f1b1b7\" (UID: \"12e9064a-baeb-44e6-b8cd-4dc046e75efe\") " pod="opendatahub/test-curl-7671-f1b1b7"
Sep 11 14:59:51.978613 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.978576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqlx2\" (UniqueName: \"kubernetes.io/projected/12e9064a-baeb-44e6-b8cd-4dc046e75efe-kube-api-access-xqlx2\") pod \"test-curl-7671-f1b1b7\" (UID: \"12e9064a-baeb-44e6-b8cd-4dc046e75efe\") " pod="opendatahub/test-curl-7671-f1b1b7"
Sep 11 14:59:51.987579 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:51.987553    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqlx2\" (UniqueName: \"kubernetes.io/projected/12e9064a-baeb-44e6-b8cd-4dc046e75efe-kube-api-access-xqlx2\") pod \"test-curl-7671-f1b1b7\" (UID: \"12e9064a-baeb-44e6-b8cd-4dc046e75efe\") " pod="opendatahub/test-curl-7671-f1b1b7"
Sep 11 14:59:52.063547 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.063476    2559 generic.go:356] "Generic (PLEG): container finished" podID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerID="b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf" exitCode=0
Sep 11 14:59:52.063547 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.063535    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" event={"ID":"81c2c16e-25b6-4584-9190-1fd68da51f25","Type":"ContainerDied","Data":"b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf"}
Sep 11 14:59:52.064083 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.064065    2559 scope.go:117] "RemoveContainer" containerID="b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf"
Sep 11 14:59:52.065065 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.065043    2559 generic.go:356] "Generic (PLEG): container finished" podID="efa1ce73-a37c-47c2-a63c-3b5978be0c4f" containerID="2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306" exitCode=137
Sep 11 14:59:52.065159 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.065140    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-db9ec5"
Sep 11 14:59:52.065229 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.065157    2559 scope.go:117] "RemoveContainer" containerID="2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306"
Sep 11 14:59:52.065426 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:52.065409    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7671-f506bf" container="test-curl-7671-f506bf"
Sep 11 14:59:52.065538 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.065425    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:59:52.065538 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.065522    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:59:52.065659 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.065571    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:59:52.075368 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.075349    2559 scope.go:117] "RemoveContainer" containerID="2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306"
Sep 11 14:59:52.075680 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:52.075658    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306\": container with ID starting with 2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306 not found: ID does not exist" containerID="2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306"
Sep 11 14:59:52.075758 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.075687    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306"} err="failed to get container status \"2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306\": rpc error: code = NotFound desc = could not find container \"2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306\": container with ID starting with 2668624da66301840a3f2975164e82ad31777b385a12ec818cd662ea07afc306 not found: ID does not exist"
Sep 11 14:59:52.085117 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:52.085094    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-db9ec5\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="efa1ce73-a37c-47c2-a63c-3b5978be0c4f" pod="opendatahub/test-curl-7671-db9ec5"
Sep 11 14:59:52.101253 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.101215    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-f1b1b7"
Sep 11 14:59:52.180275 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.180244    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efa1ce73-a37c-47c2-a63c-3b5978be0c4f" path="/var/lib/kubelet/pods/efa1ce73-a37c-47c2-a63c-3b5978be0c4f/volumes"
Sep 11 14:59:52.378704 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.378661    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-f1b1b7"]
Sep 11 14:59:52.380783 ip-10-0-128-223 kubenswrapper[2559]: W0911 14:59:52.380751    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12e9064a_baeb_44e6_b8cd_4dc046e75efe.slice/crio-e6f7cbd838049d32be75e408fa8e65901dc23482e3499a7f206dcd253b628ca0 WatchSource:0}: Error finding container e6f7cbd838049d32be75e408fa8e65901dc23482e3499a7f206dcd253b628ca0: Status 404 returned error can't find the container with id e6f7cbd838049d32be75e408fa8e65901dc23482e3499a7f206dcd253b628ca0
Sep 11 14:59:52.384873 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:52.384846    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:53.052603 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.052562    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:53.076063 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.075489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-f1b1b7" event={"ID":"12e9064a-baeb-44e6-b8cd-4dc046e75efe","Type":"ContainerStarted","Data":"e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df"}
Sep 11 14:59:53.076063 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.075539    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-f1b1b7" event={"ID":"12e9064a-baeb-44e6-b8cd-4dc046e75efe","Type":"ContainerStarted","Data":"e6f7cbd838049d32be75e408fa8e65901dc23482e3499a7f206dcd253b628ca0"}
Sep 11 14:59:53.078522 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.078490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" event={"ID":"81c2c16e-25b6-4584-9190-1fd68da51f25","Type":"ContainerStarted","Data":"7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7"}
Sep 11 14:59:53.081160 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.081127    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7671-f506bf" podUID="a2a4759b-8c66-418f-b071-147ed25ca881" containerName="test-curl-7671-f506bf" containerID="cri-o://1278120915b167ee48a9238c07c85af69a5a67060546001fd34fbcffb0997363" gracePeriod=2
Sep 11 14:59:53.093923 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.093871    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7671-f1b1b7" podStartSLOduration=2.09385246 podStartE2EDuration="2.09385246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:53.091666892 +0000 UTC m=+1199.414295263" watchObservedRunningTime="2026-09-11 14:59:53.09385246 +0000 UTC m=+1199.416480830"
Sep 11 14:59:53.196290 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.196251    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:53.196593 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.196567    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 11 14:59:53.196685 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.196663    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 11 14:59:53.248598 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.248578    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-63b94f"
Sep 11 14:59:53.294249 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.294218    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87ch7\" (UniqueName: \"kubernetes.io/projected/9e88467e-02a2-4458-b85d-69f46bf776d2-kube-api-access-87ch7\") pod \"9e88467e-02a2-4458-b85d-69f46bf776d2\" (UID: \"9e88467e-02a2-4458-b85d-69f46bf776d2\") "
Sep 11 14:59:53.296393 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.296363    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e88467e-02a2-4458-b85d-69f46bf776d2-kube-api-access-87ch7" (OuterVolumeSpecName: "kube-api-access-87ch7") pod "9e88467e-02a2-4458-b85d-69f46bf776d2" (UID: "9e88467e-02a2-4458-b85d-69f46bf776d2"). InnerVolumeSpecName "kube-api-access-87ch7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:53.395705 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.395664    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87ch7\" (UniqueName: \"kubernetes.io/projected/9e88467e-02a2-4458-b85d-69f46bf776d2-kube-api-access-87ch7\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:53.742437 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.742399    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7671-f1b1b7"]
Sep 11 14:59:53.748096 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.748064    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7671-f1b1b7"]
Sep 11 14:59:53.799158 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.799117    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"]
Sep 11 14:59:53.800556 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.800510    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc" podUID="a6129071-08b5-4e39-b0a9-340911efeaa7" containerName="maas-api" containerID="cri-o://ef5c892707cee958f0a6d3a3a76bd2ad2182b5a3d13b166a32201572229c7682" gracePeriod=30
Sep 11 14:59:53.822433 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.822380    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:53.905108 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.905060    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:53.991992 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:53.991952    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:54.091997 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.091969    2559 generic.go:356] "Generic (PLEG): container finished" podID="a6129071-08b5-4e39-b0a9-340911efeaa7" containerID="ef5c892707cee958f0a6d3a3a76bd2ad2182b5a3d13b166a32201572229c7682" exitCode=0
Sep 11 14:59:54.092401 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.092047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc" event={"ID":"a6129071-08b5-4e39-b0a9-340911efeaa7","Type":"ContainerDied","Data":"ef5c892707cee958f0a6d3a3a76bd2ad2182b5a3d13b166a32201572229c7682"}
Sep 11 14:59:54.092401 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.092336    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:54.093578 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.093554    2559 generic.go:356] "Generic (PLEG): container finished" podID="9e88467e-02a2-4458-b85d-69f46bf776d2" containerID="24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690" exitCode=137
Sep 11 14:59:54.093695 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.093609    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-63b94f"
Sep 11 14:59:54.093695 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.093653    2559 scope.go:117] "RemoveContainer" containerID="24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690"
Sep 11 14:59:54.094162 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:54.093965    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7671-f1b1b7" container="test-curl-7671-f1b1b7"
Sep 11 14:59:54.105600 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.105578    2559 scope.go:117] "RemoveContainer" containerID="24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690"
Sep 11 14:59:54.105968 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:54.105935    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690\": container with ID starting with 24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690 not found: ID does not exist" containerID="24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690"
Sep 11 14:59:54.106083 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.105966    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690"} err="failed to get container status \"24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690\": rpc error: code = NotFound desc = could not find container \"24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690\": container with ID starting with 24035059c2cfa412ba32eaf2d3b2b25ee7aae8fd178ddb7629c0faf46b84b690 not found: ID does not exist"
Sep 11 14:59:54.108392 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:54.108364    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-63b94f\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="9e88467e-02a2-4458-b85d-69f46bf776d2" pod="opendatahub/test-curl-7671-63b94f"
Sep 11 14:59:54.109949 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.109925    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:59:54.145826 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:54.132524    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-63b94f\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="9e88467e-02a2-4458-b85d-69f46bf776d2" pod="opendatahub/test-curl-7671-63b94f"
Sep 11 14:59:54.185478 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.180837    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:54.198488 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:54.197754    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-63b94f\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="9e88467e-02a2-4458-b85d-69f46bf776d2" pod="opendatahub/test-curl-7671-63b94f"
Sep 11 14:59:54.198488 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.198319    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 11 14:59:54.198488 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.198370    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 11 14:59:54.200208 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.200177    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e88467e-02a2-4458-b85d-69f46bf776d2" path="/var/lib/kubelet/pods/9e88467e-02a2-4458-b85d-69f46bf776d2/volumes"
Sep 11 14:59:54.203767 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.203629    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-metrics-certs\") pod \"a6129071-08b5-4e39-b0a9-340911efeaa7\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") "
Sep 11 14:59:54.203767 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.203672    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6129071-08b5-4e39-b0a9-340911efeaa7-openshift-service-ca\") pod \"a6129071-08b5-4e39-b0a9-340911efeaa7\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") "
Sep 11 14:59:54.203767 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.203707    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwqq6\" (UniqueName: \"kubernetes.io/projected/a6129071-08b5-4e39-b0a9-340911efeaa7-kube-api-access-pwqq6\") pod \"a6129071-08b5-4e39-b0a9-340911efeaa7\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") "
Sep 11 14:59:54.204057 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.203783    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-maas-api-tls\") pod \"a6129071-08b5-4e39-b0a9-340911efeaa7\" (UID: \"a6129071-08b5-4e39-b0a9-340911efeaa7\") "
Sep 11 14:59:54.204115 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.204095    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6129071-08b5-4e39-b0a9-340911efeaa7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6129071-08b5-4e39-b0a9-340911efeaa7" (UID: "a6129071-08b5-4e39-b0a9-340911efeaa7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:59:54.204209 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.204180    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6129071-08b5-4e39-b0a9-340911efeaa7-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:54.206835 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.206810    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a6129071-08b5-4e39-b0a9-340911efeaa7" (UID: "a6129071-08b5-4e39-b0a9-340911efeaa7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:59:54.206943 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.206904    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6129071-08b5-4e39-b0a9-340911efeaa7" (UID: "a6129071-08b5-4e39-b0a9-340911efeaa7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:59:54.207156 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.207135    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6129071-08b5-4e39-b0a9-340911efeaa7-kube-api-access-pwqq6" (OuterVolumeSpecName: "kube-api-access-pwqq6") pod "a6129071-08b5-4e39-b0a9-340911efeaa7" (UID: "a6129071-08b5-4e39-b0a9-340911efeaa7"). InnerVolumeSpecName "kube-api-access-pwqq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:54.283352 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.283261    2559 scope.go:117] "RemoveContainer" containerID="ef5c892707cee958f0a6d3a3a76bd2ad2182b5a3d13b166a32201572229c7682"
Sep 11 14:59:54.305409 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.305381    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwqq6\" (UniqueName: \"kubernetes.io/projected/a6129071-08b5-4e39-b0a9-340911efeaa7-kube-api-access-pwqq6\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:54.305409 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.305409    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:54.305594 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.305419    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6129071-08b5-4e39-b0a9-340911efeaa7-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:54.353770 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.353732    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:54.483929 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.483888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:54.583757 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.583655    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:54.688261 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:54.688229    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:55.099726 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.099699    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc" event={"ID":"a6129071-08b5-4e39-b0a9-340911efeaa7","Type":"ContainerDied","Data":"150d6644adb81532d1faba1b2d3f07537876e428f9a9d98c07e76260720937f7"}
Sep 11 14:59:55.101435 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.101410    2559 generic.go:356] "Generic (PLEG): container finished" podID="a2a4759b-8c66-418f-b071-147ed25ca881" containerID="1278120915b167ee48a9238c07c85af69a5a67060546001fd34fbcffb0997363" exitCode=137
Sep 11 14:59:55.101545 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.101479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"
Sep 11 14:59:55.101692 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.101668    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7671-f1b1b7" podUID="12e9064a-baeb-44e6-b8cd-4dc046e75efe" containerName="test-curl-7671-f1b1b7" containerID="cri-o://e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df" gracePeriod=2
Sep 11 14:59:55.150921 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.150888    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"]
Sep 11 14:59:55.175984 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.175254    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90d6f5-56bf55b87c-qhrwc"]
Sep 11 14:59:55.197253 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.197221    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 11 14:59:55.197403 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.197277    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 11 14:59:55.247546 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.247509    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:55.253620 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.253601    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-f506bf"
Sep 11 14:59:55.316136 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.316106    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpmqh\" (UniqueName: \"kubernetes.io/projected/a2a4759b-8c66-418f-b071-147ed25ca881-kube-api-access-rpmqh\") pod \"a2a4759b-8c66-418f-b071-147ed25ca881\" (UID: \"a2a4759b-8c66-418f-b071-147ed25ca881\") "
Sep 11 14:59:55.318271 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.318245    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a4759b-8c66-418f-b071-147ed25ca881-kube-api-access-rpmqh" (OuterVolumeSpecName: "kube-api-access-rpmqh") pod "a2a4759b-8c66-418f-b071-147ed25ca881" (UID: "a2a4759b-8c66-418f-b071-147ed25ca881"). InnerVolumeSpecName "kube-api-access-rpmqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:55.417148 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.417057    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpmqh\" (UniqueName: \"kubernetes.io/projected/a2a4759b-8c66-418f-b071-147ed25ca881-kube-api-access-rpmqh\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:55.579349 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:55.579317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:56.107394 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:56.107369    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-f506bf"
Sep 11 14:59:56.107807 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:56.107413    2559 scope.go:117] "RemoveContainer" containerID="1278120915b167ee48a9238c07c85af69a5a67060546001fd34fbcffb0997363"
Sep 11 14:59:56.117768 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:56.117741    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-f506bf\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="a2a4759b-8c66-418f-b071-147ed25ca881" pod="opendatahub/test-curl-7671-f506bf"
Sep 11 14:59:56.185969 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:56.185920    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2a4759b-8c66-418f-b071-147ed25ca881" path="/var/lib/kubelet/pods/a2a4759b-8c66-418f-b071-147ed25ca881/volumes"
Sep 11 14:59:56.186582 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:56.186561    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6129071-08b5-4e39-b0a9-340911efeaa7" path="/var/lib/kubelet/pods/a6129071-08b5-4e39-b0a9-340911efeaa7/volumes"
Sep 11 14:59:56.196567 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:56.196535    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 11 14:59:56.196700 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:56.196583    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 11 14:59:56.241557 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:56.241517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:57.196994 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:57.196958    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 11 14:59:57.197364 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:57.197020    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 11 14:59:57.204991 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:57.204938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:57.329577 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:57.329549    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-f1b1b7"
Sep 11 14:59:57.439826 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:57.439739    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqlx2\" (UniqueName: \"kubernetes.io/projected/12e9064a-baeb-44e6-b8cd-4dc046e75efe-kube-api-access-xqlx2\") pod \"12e9064a-baeb-44e6-b8cd-4dc046e75efe\" (UID: \"12e9064a-baeb-44e6-b8cd-4dc046e75efe\") "
Sep 11 14:59:57.441827 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:57.441803    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e9064a-baeb-44e6-b8cd-4dc046e75efe-kube-api-access-xqlx2" (OuterVolumeSpecName: "kube-api-access-xqlx2") pod "12e9064a-baeb-44e6-b8cd-4dc046e75efe" (UID: "12e9064a-baeb-44e6-b8cd-4dc046e75efe"). InnerVolumeSpecName "kube-api-access-xqlx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:57.540561 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:57.540527    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqlx2\" (UniqueName: \"kubernetes.io/projected/12e9064a-baeb-44e6-b8cd-4dc046e75efe-kube-api-access-xqlx2\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 14:59:57.586365 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:57.586323    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:57.686479 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:57.685880    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:58.029621 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.029583    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:58.119347 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.119311    2559 generic.go:356] "Generic (PLEG): container finished" podID="12e9064a-baeb-44e6-b8cd-4dc046e75efe" containerID="e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df" exitCode=137
Sep 11 14:59:58.119546 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.119356    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-f1b1b7"
Sep 11 14:59:58.119546 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.119414    2559 scope.go:117] "RemoveContainer" containerID="e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df"
Sep 11 14:59:58.128190 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.128167    2559 scope.go:117] "RemoveContainer" containerID="e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df"
Sep 11 14:59:58.128522 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:58.128502    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df\": container with ID starting with e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df not found: ID does not exist" containerID="e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df"
Sep 11 14:59:58.128624 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.128530    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df"} err="failed to get container status \"e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df\": rpc error: code = NotFound desc = could not find container \"e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df\": container with ID starting with e4eea9c20b59a99ce4a7156af15d25320cf24fa443572066dade9eb1909d75df not found: ID does not exist"
Sep 11 14:59:58.131003 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:58.130979    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-f1b1b7\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="12e9064a-baeb-44e6-b8cd-4dc046e75efe" pod="opendatahub/test-curl-7671-f1b1b7"
Sep 11 14:59:58.184717 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.184677    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e9064a-baeb-44e6-b8cd-4dc046e75efe" path="/var/lib/kubelet/pods/12e9064a-baeb-44e6-b8cd-4dc046e75efe/volumes"
Sep 11 14:59:58.259274 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.259239    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:58.259752 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.259730    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:58.260912 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.260894    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 14:59:58.975949 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.975919    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"]
Sep 11 14:59:58.976428 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.976403    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e9064a-baeb-44e6-b8cd-4dc046e75efe" containerName="test-curl-7671-f1b1b7"
Sep 11 14:59:58.976428 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.976427    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e9064a-baeb-44e6-b8cd-4dc046e75efe" containerName="test-curl-7671-f1b1b7"
Sep 11 14:59:58.976659 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.976470    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6129071-08b5-4e39-b0a9-340911efeaa7" containerName="maas-api"
Sep 11 14:59:58.976659 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.976479    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6129071-08b5-4e39-b0a9-340911efeaa7" containerName="maas-api"
Sep 11 14:59:58.976659 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.976558    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6129071-08b5-4e39-b0a9-340911efeaa7" containerName="maas-api"
Sep 11 14:59:58.976659 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.976571    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e9064a-baeb-44e6-b8cd-4dc046e75efe" containerName="test-curl-7671-f1b1b7"
Sep 11 14:59:58.981184 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.981160    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:58.984015 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.983985    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a843ae03-serving-cert\""
Sep 11 14:59:58.991076 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:58.991040    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"]
Sep 11 14:59:59.007249 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.007217    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:59.057101 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.057072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-metrics-certs\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.057232 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.057105    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbda0a30-75bb-475d-aacc-80bf75dafbce-openshift-service-ca\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.057232 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.057130    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-maas-api-tls\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.057347 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.057248    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6cw\" (UniqueName: \"kubernetes.io/projected/fbda0a30-75bb-475d-aacc-80bf75dafbce-kube-api-access-tc6cw\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.073809 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.073785    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:59.158118 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.158084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-metrics-certs\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.158612 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.158133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbda0a30-75bb-475d-aacc-80bf75dafbce-openshift-service-ca\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.158612 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.158169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-maas-api-tls\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.158612 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.158267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6cw\" (UniqueName: \"kubernetes.io/projected/fbda0a30-75bb-475d-aacc-80bf75dafbce-kube-api-access-tc6cw\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.158612 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:59.158310    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-serving-cert: secret "maas-api-e2e-mt-a843ae03-serving-cert" not found
Sep 11 14:59:59.158612 ip-10-0-128-223 kubenswrapper[2559]: E0911 14:59:59.158379    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-maas-api-tls podName:fbda0a30-75bb-475d-aacc-80bf75dafbce nodeName:}" failed. No retries permitted until 2026-09-11 14:59:59.658358076 +0000 UTC m=+1205.980986440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-maas-api-tls") pod "maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz" (UID: "fbda0a30-75bb-475d-aacc-80bf75dafbce") : secret "maas-api-e2e-mt-a843ae03-serving-cert" not found
Sep 11 14:59:59.158971 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.158944    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbda0a30-75bb-475d-aacc-80bf75dafbce-openshift-service-ca\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.160776 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.160754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-metrics-certs\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.167217 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.167197    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6cw\" (UniqueName: \"kubernetes.io/projected/fbda0a30-75bb-475d-aacc-80bf75dafbce-kube-api-access-tc6cw\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.332366 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.332279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:59.467541 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.467501    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"]
Sep 11 14:59:59.562792 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.562756    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:59.643706 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.643609    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 14:59:59.663919 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.663886    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-maas-api-tls\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.666290 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.666264    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-maas-api-tls\") pod \"maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.893019 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.892989    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 14:59:59.903814 ip-10-0-128-223 kubenswrapper[2559]: I0911 14:59:59.903770    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:00.147946 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.147870    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s"]
Sep 11 15:00:00.152348 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.152328    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k"]
Sep 11 15:00:00.152449 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.152432    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s"
Sep 11 15:00:00.154827 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.154806    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\""
Sep 11 15:00:00.155852 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.155828    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k"
Sep 11 15:00:00.170079 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.169906    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s"
Sep 11 15:00:00.172596 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.172572    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s"]
Sep 11 15:00:00.173749 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.173551    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k"]
Sep 11 15:00:00.199146 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:00:00.199104    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbda0a30_75bb_475d_aacc_80bf75dafbce.slice/crio-b2c0826ddfd4f4f4b492ce499311d928171b4122149d86306a79c35e20559dcf WatchSource:0}: Error finding container b2c0826ddfd4f4f4b492ce499311d928171b4122149d86306a79c35e20559dcf: Status 404 returned error can't find the container with id b2c0826ddfd4f4f4b492ce499311d928171b4122149d86306a79c35e20559dcf
Sep 11 15:00:00.204245 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.204197    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"]
Sep 11 15:00:00.216582 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.216556    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k"
Sep 11 15:00:00.470205 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.470179    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s"]
Sep 11 15:00:00.472055 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:00:00.472027    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c5b812_2b71_4dc5_9efb_42d59c68993c.slice/crio-61601d4d7d1d8c6721ba2da8b8257b510c30c380dc7380d4b5941a752034bb0a WatchSource:0}: Error finding container 61601d4d7d1d8c6721ba2da8b8257b510c30c380dc7380d4b5941a752034bb0a: Status 404 returned error can't find the container with id 61601d4d7d1d8c6721ba2da8b8257b510c30c380dc7380d4b5941a752034bb0a
Sep 11 15:00:00.498348 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.498325    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k"]
Sep 11 15:00:00.499664 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:00:00.499642    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55a9325_2818_4247_b865_d665347f86e4.slice/crio-1ee57412b602a7a67d2bb31904f767ab4cd4901d60fcb2c0bdcd081632b8fb81 WatchSource:0}: Error finding container 1ee57412b602a7a67d2bb31904f767ab4cd4901d60fcb2c0bdcd081632b8fb81: Status 404 returned error can't find the container with id 1ee57412b602a7a67d2bb31904f767ab4cd4901d60fcb2c0bdcd081632b8fb81
Sep 11 15:00:00.679430 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:00.679395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:01.136997 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:01.136961    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k" event={"ID":"b55a9325-2818-4247-b865-d665347f86e4","Type":"ContainerStarted","Data":"1ee57412b602a7a67d2bb31904f767ab4cd4901d60fcb2c0bdcd081632b8fb81"}
Sep 11 15:00:01.138128 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:01.138097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s" event={"ID":"c5c5b812-2b71-4dc5-9efb-42d59c68993c","Type":"ContainerStarted","Data":"61601d4d7d1d8c6721ba2da8b8257b510c30c380dc7380d4b5941a752034bb0a"}
Sep 11 15:00:01.139755 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:01.139731    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz" event={"ID":"fbda0a30-75bb-475d-aacc-80bf75dafbce","Type":"ContainerStarted","Data":"ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22"}
Sep 11 15:00:01.139903 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:01.139777    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz" event={"ID":"fbda0a30-75bb-475d-aacc-80bf75dafbce","Type":"ContainerStarted","Data":"b2c0826ddfd4f4f4b492ce499311d928171b4122149d86306a79c35e20559dcf"}
Sep 11 15:00:01.139903 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:01.139843    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 15:00:01.160211 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:01.160168    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz" podStartSLOduration=2.399967106 podStartE2EDuration="3.16015286s" totalImagesPullingTime="760.185754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:00:01.159878579 +0000 UTC m=+1207.482506948" watchObservedRunningTime="2026-09-11 15:00:01.16015286 +0000 UTC m=+1207.482781229"
Sep 11 15:00:01.543052 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:01.543010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:02.534191 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:02.534146    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:04.136016 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:04.135981    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:00:04.158234 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:04.158200    2559 generic.go:356] "Generic (PLEG): container finished" podID="b55a9325-2818-4247-b865-d665347f86e4" containerID="2d2de6e8a5d39dc72c4f1a519a839a39da06560df574befcb0d213c8fc4f3b2f" exitCode=0
Sep 11 15:00:04.158422 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:04.158298    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k" event={"ID":"b55a9325-2818-4247-b865-d665347f86e4","Type":"ContainerDied","Data":"2d2de6e8a5d39dc72c4f1a519a839a39da06560df574befcb0d213c8fc4f3b2f"}
Sep 11 15:00:04.160037 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:04.160016    2559 generic.go:356] "Generic (PLEG): container finished" podID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerID="5332553b0652e32d26eca09de7d871366250847132bb6b7ecdd8dd7a84f923c8" exitCode=7
Sep 11 15:00:04.160157 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:04.160086    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s" event={"ID":"c5c5b812-2b71-4dc5-9efb-42d59c68993c","Type":"ContainerDied","Data":"5332553b0652e32d26eca09de7d871366250847132bb6b7ecdd8dd7a84f923c8"}
Sep 11 15:00:04.160249 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:04.160233    2559 scope.go:117] "RemoveContainer" containerID="5332553b0652e32d26eca09de7d871366250847132bb6b7ecdd8dd7a84f923c8"
Sep 11 15:00:05.168587 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:05.168496    2559 generic.go:356] "Generic (PLEG): container finished" podID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerID="dce9dab0aa7ca451113fd61f24f9673f3080125b9b3d384292923c9133e4c62d" exitCode=7
Sep 11 15:00:05.169047 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:05.168580    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s" event={"ID":"c5c5b812-2b71-4dc5-9efb-42d59c68993c","Type":"ContainerDied","Data":"dce9dab0aa7ca451113fd61f24f9673f3080125b9b3d384292923c9133e4c62d"}
Sep 11 15:00:05.169047 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:05.168647    2559 scope.go:117] "RemoveContainer" containerID="5332553b0652e32d26eca09de7d871366250847132bb6b7ecdd8dd7a84f923c8"
Sep 11 15:00:05.169047 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:05.168837    2559 scope.go:117] "RemoveContainer" containerID="dce9dab0aa7ca451113fd61f24f9673f3080125b9b3d384292923c9133e4c62d"
Sep 11 15:00:05.169559 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:00:05.169112    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s_odh-ai-gateway-infra(c5c5b812-2b71-4dc5-9efb-42d59c68993c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c"
Sep 11 15:00:05.310848 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:05.310819    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k"
Sep 11 15:00:06.176045 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:06.176013    2559 scope.go:117] "RemoveContainer" containerID="dce9dab0aa7ca451113fd61f24f9673f3080125b9b3d384292923c9133e4c62d"
Sep 11 15:00:06.176556 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:00:06.176288    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s_odh-ai-gateway-infra(c5c5b812-2b71-4dc5-9efb-42d59c68993c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c"
Sep 11 15:00:06.177323 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:06.177302    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k"
Sep 11 15:00:06.182020 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:06.181995    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k" event={"ID":"b55a9325-2818-4247-b865-d665347f86e4","Type":"ContainerDied","Data":"1ee57412b602a7a67d2bb31904f767ab4cd4901d60fcb2c0bdcd081632b8fb81"}
Sep 11 15:00:06.182144 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:06.182035    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ee57412b602a7a67d2bb31904f767ab4cd4901d60fcb2c0bdcd081632b8fb81"
Sep 11 15:00:07.152619 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:07.152585    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 15:00:11.874317 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:11.874278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:14.486396 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.486357    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj" podUID="598c5590-02d6-41ec-8e30-12043837e367" containerName="main" containerID="cri-o://18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8" gracePeriod=45
Sep 11 15:00:14.740140 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.740075    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 15:00:14.839673 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.839638    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-dshm\") pod \"598c5590-02d6-41ec-8e30-12043837e367\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") "
Sep 11 15:00:14.839847 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.839679    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598c5590-02d6-41ec-8e30-12043837e367-tls-certs\") pod \"598c5590-02d6-41ec-8e30-12043837e367\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") "
Sep 11 15:00:14.839847 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.839706    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-home\") pod \"598c5590-02d6-41ec-8e30-12043837e367\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") "
Sep 11 15:00:14.839847 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.839732    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-tmp-dir\") pod \"598c5590-02d6-41ec-8e30-12043837e367\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") "
Sep 11 15:00:14.839847 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.839828    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-model-cache\") pod \"598c5590-02d6-41ec-8e30-12043837e367\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") "
Sep 11 15:00:14.840066 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.839936    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgltk\" (UniqueName: \"kubernetes.io/projected/598c5590-02d6-41ec-8e30-12043837e367-kube-api-access-tgltk\") pod \"598c5590-02d6-41ec-8e30-12043837e367\" (UID: \"598c5590-02d6-41ec-8e30-12043837e367\") "
Sep 11 15:00:14.840066 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.839998    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-home" (OuterVolumeSpecName: "home") pod "598c5590-02d6-41ec-8e30-12043837e367" (UID: "598c5590-02d6-41ec-8e30-12043837e367"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:00:14.840183 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.840111    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "598c5590-02d6-41ec-8e30-12043837e367" (UID: "598c5590-02d6-41ec-8e30-12043837e367"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:00:14.840183 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.840159    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-model-cache" (OuterVolumeSpecName: "model-cache") pod "598c5590-02d6-41ec-8e30-12043837e367" (UID: "598c5590-02d6-41ec-8e30-12043837e367"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:00:14.840359 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.840334    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:00:14.840483 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.840362    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:00:14.840483 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.840378    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:00:14.841877 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.841852    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-dshm" (OuterVolumeSpecName: "dshm") pod "598c5590-02d6-41ec-8e30-12043837e367" (UID: "598c5590-02d6-41ec-8e30-12043837e367"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:00:14.841962 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.841913    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598c5590-02d6-41ec-8e30-12043837e367-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "598c5590-02d6-41ec-8e30-12043837e367" (UID: "598c5590-02d6-41ec-8e30-12043837e367"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:00:14.841962 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.841918    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598c5590-02d6-41ec-8e30-12043837e367-kube-api-access-tgltk" (OuterVolumeSpecName: "kube-api-access-tgltk") pod "598c5590-02d6-41ec-8e30-12043837e367" (UID: "598c5590-02d6-41ec-8e30-12043837e367"). InnerVolumeSpecName "kube-api-access-tgltk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:00:14.941338 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.941298    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgltk\" (UniqueName: \"kubernetes.io/projected/598c5590-02d6-41ec-8e30-12043837e367-kube-api-access-tgltk\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:00:14.941338 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.941331    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598c5590-02d6-41ec-8e30-12043837e367-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:00:14.941338 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:14.941341    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598c5590-02d6-41ec-8e30-12043837e367-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:00:15.230682 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:15.230647    2559 generic.go:356] "Generic (PLEG): container finished" podID="598c5590-02d6-41ec-8e30-12043837e367" containerID="18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8" exitCode=0
Sep 11 15:00:15.230851 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:15.230687    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"
Sep 11 15:00:15.230851 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:15.230708    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj" event={"ID":"598c5590-02d6-41ec-8e30-12043837e367","Type":"ContainerDied","Data":"18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8"}
Sep 11 15:00:15.230851 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:15.230745    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj" event={"ID":"598c5590-02d6-41ec-8e30-12043837e367","Type":"ContainerDied","Data":"0f42dc93e8c8e66ff4ed407084672f42f36c2c0dd753ad215ab67c5f06aaa241"}
Sep 11 15:00:15.230851 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:15.230765    2559 scope.go:117] "RemoveContainer" containerID="18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8"
Sep 11 15:00:15.247547 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:15.247528    2559 scope.go:117] "RemoveContainer" containerID="18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8"
Sep 11 15:00:15.247839 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:00:15.247814    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8\": container with ID starting with 18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8 not found: ID does not exist" containerID="18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8"
Sep 11 15:00:15.247911 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:15.247846    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8"} err="failed to get container status \"18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8\": rpc error: code = NotFound desc = could not find container \"18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8\": container with ID starting with 18cea16ec123be4f1eaa15f6afdbabb1f2bc5e23ee01e3e8b690b6544011c5d8 not found: ID does not exist"
Sep 11 15:00:15.259741 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:15.259619    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"]
Sep 11 15:00:15.261977 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:15.261953    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-90d6f5/test-model-90d6f5-kserve-b5df94f69-tltlj"]
Sep 11 15:00:16.180582 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:16.180544    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="598c5590-02d6-41ec-8e30-12043837e367" path="/var/lib/kubelet/pods/598c5590-02d6-41ec-8e30-12043837e367/volumes"
Sep 11 15:00:17.178588 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:17.178556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:20.007602 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:20.007566    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:21.174101 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:21.174076    2559 scope.go:117] "RemoveContainer" containerID="dce9dab0aa7ca451113fd61f24f9673f3080125b9b3d384292923c9133e4c62d"
Sep 11 15:00:21.514157 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:21.514128    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:22.254007 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:22.253972    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:22.270597 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:22.270567    2559 generic.go:356] "Generic (PLEG): container finished" podID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerID="837be2e91986db4bdd5c01acff38839c91e1fb8b8a1ddc9bdf42506e1f59de76" exitCode=0
Sep 11 15:00:22.270753 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:22.270625    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s" event={"ID":"c5c5b812-2b71-4dc5-9efb-42d59c68993c","Type":"ContainerDied","Data":"837be2e91986db4bdd5c01acff38839c91e1fb8b8a1ddc9bdf42506e1f59de76"}
Sep 11 15:00:22.270753 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:22.270671    2559 scope.go:117] "RemoveContainer" containerID="dce9dab0aa7ca451113fd61f24f9673f3080125b9b3d384292923c9133e4c62d"
Sep 11 15:00:22.743223 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:22.743184    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:23.241030 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:23.241001    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:23.319137 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:23.319105    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s"]
Sep 11 15:00:23.412918 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:23.412891    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s"
Sep 11 15:00:24.124079 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.124037    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:24.285543 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.285509    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s"
Sep 11 15:00:24.285718 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.285560    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s" event={"ID":"c5c5b812-2b71-4dc5-9efb-42d59c68993c","Type":"ContainerDied","Data":"61601d4d7d1d8c6721ba2da8b8257b510c30c380dc7380d4b5941a752034bb0a"}
Sep 11 15:00:24.285718 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.285605    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61601d4d7d1d8c6721ba2da8b8257b510c30c380dc7380d4b5941a752034bb0a"
Sep 11 15:00:24.316211 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.316178    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s"]
Sep 11 15:00:24.324211 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.324185    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a843ae03-29818980-d2f4s"]
Sep 11 15:00:24.771844 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.771811    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"]
Sep 11 15:00:24.772229 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772216    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerName="cleanup"
Sep 11 15:00:24.772295 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772230    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerName="cleanup"
Sep 11 15:00:24.772295 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772242    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598c5590-02d6-41ec-8e30-12043837e367" containerName="main"
Sep 11 15:00:24.772295 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772248    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="598c5590-02d6-41ec-8e30-12043837e367" containerName="main"
Sep 11 15:00:24.772295 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772265    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b55a9325-2818-4247-b865-d665347f86e4" containerName="cleanup"
Sep 11 15:00:24.772295 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772271    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b55a9325-2818-4247-b865-d665347f86e4" containerName="cleanup"
Sep 11 15:00:24.772295 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772284    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerName="cleanup"
Sep 11 15:00:24.772295 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772290    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerName="cleanup"
Sep 11 15:00:24.772687 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772350    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerName="cleanup"
Sep 11 15:00:24.772687 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772357    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerName="cleanup"
Sep 11 15:00:24.772687 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772363    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b55a9325-2818-4247-b865-d665347f86e4" containerName="cleanup"
Sep 11 15:00:24.772687 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772371    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="598c5590-02d6-41ec-8e30-12043837e367" containerName="main"
Sep 11 15:00:24.772687 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.772376    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerName="cleanup"
Sep 11 15:00:24.775759 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.775738    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.779638 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.779377    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-cd2cc1\"/\"openshift-service-ca.crt\""
Sep 11 15:00:24.779638 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.779422    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-cd2cc1\"/\"default-dockercfg-wxz6k\""
Sep 11 15:00:24.779638 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.779442    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-cd2cc1\"/\"noten-model-cd2cc1-kserve-self-signed-certs\""
Sep 11 15:00:24.779638 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.779508    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-cd2cc1\"/\"kube-root-ca.crt\""
Sep 11 15:00:24.787000 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.786977    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"]
Sep 11 15:00:24.803307 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.803261    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:24.842769 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.842740    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-model-cache\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.842939 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.842775    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tmp-dir\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.842939 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.842840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-dshm\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.842939 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.842898    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-home\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.842939 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.842932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tls-certs\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.843157 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.843016    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpjpp\" (UniqueName: \"kubernetes.io/projected/b1e0e918-9125-4b5a-8d91-b811a6fd7172-kube-api-access-fpjpp\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.931484 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.931434    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:24.944215 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.944185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjpp\" (UniqueName: \"kubernetes.io/projected/b1e0e918-9125-4b5a-8d91-b811a6fd7172-kube-api-access-fpjpp\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.944394 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.944251    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-model-cache\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.944394 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.944272    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tmp-dir\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.944394 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.944307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-dshm\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.944394 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.944360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-home\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.944394 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.944382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tls-certs\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.944789 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.944768    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-model-cache\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.944853 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.944816    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tmp-dir\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.944853 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.944824    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-home\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.946577 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.946560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-dshm\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.946940 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.946923    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tls-certs\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:24.953530 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:24.953512    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpjpp\" (UniqueName: \"kubernetes.io/projected/b1e0e918-9125-4b5a-8d91-b811a6fd7172-kube-api-access-fpjpp\") pod \"noten-model-cd2cc1-kserve-8d578c7d7-wgl8q\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") " pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:25.074915 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:25.074827    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:25.085768 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:25.085741    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:25.175405 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:25.175374    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:25.359559 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:25.359533    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"]
Sep 11 15:00:25.361748 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:00:25.361716    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1e0e918_9125_4b5a_8d91_b811a6fd7172.slice/crio-2508e9e90473b96fd716380505d7d4513539d9c6909f583fe10dac3be069a37b WatchSource:0}: Error finding container 2508e9e90473b96fd716380505d7d4513539d9c6909f583fe10dac3be069a37b: Status 404 returned error can't find the container with id 2508e9e90473b96fd716380505d7d4513539d9c6909f583fe10dac3be069a37b
Sep 11 15:00:25.946613 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:25.946579    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:26.187251 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:26.181117    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" path="/var/lib/kubelet/pods/c5c5b812-2b71-4dc5-9efb-42d59c68993c/volumes"
Sep 11 15:00:26.297923 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:26.297892    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q" event={"ID":"b1e0e918-9125-4b5a-8d91-b811a6fd7172","Type":"ContainerStarted","Data":"a58d672c0e9093863f3db8fef72b3308450ed964988a4742ef266b752287fd25"}
Sep 11 15:00:26.297923 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:26.297932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q" event={"ID":"b1e0e918-9125-4b5a-8d91-b811a6fd7172","Type":"ContainerStarted","Data":"2508e9e90473b96fd716380505d7d4513539d9c6909f583fe10dac3be069a37b"}
Sep 11 15:00:26.320005 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:26.319946    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q" podStartSLOduration=2.319932638 podStartE2EDuration="2.319932638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:00:26.316919469 +0000 UTC m=+1232.639547838" watchObservedRunningTime="2026-09-11 15:00:26.319932638 +0000 UTC m=+1232.642561073"
Sep 11 15:00:33.316384 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:33.316286    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:00:35.086578 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:35.086547    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:35.086991 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:35.086618    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:35.099319 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:35.099295    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:35.361580 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:35.361489    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:00:36.754878 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:36.754844    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:38.132396 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:38.132351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:38.235039 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:38.235001    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:40.409683 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:40.409651    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:41.064513 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:41.064472    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:41.163139 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:41.163100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:42.033240 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:42.033199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:42.824615 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:42.824574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:43.619967 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:43.619933    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:44.415320 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:44.415273    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:45.385943 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.385912    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"]
Sep 11 15:00:45.386337 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.386326    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerName="cleanup"
Sep 11 15:00:45.386412 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.386340    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c5b812-2b71-4dc5-9efb-42d59c68993c" containerName="cleanup"
Sep 11 15:00:45.390606 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.390587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.393315 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.393297    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-48898f-serving-cert\""
Sep 11 15:00:45.401205 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.401181    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"]
Sep 11 15:00:45.413662 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.413628    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:45.482473 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.482431    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:45.552481 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.552414    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.552666 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.552548    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c21653-ad82-4871-bca5-505fbf4c70bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.552666 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.552659    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6rdk\" (UniqueName: \"kubernetes.io/projected/c7c21653-ad82-4871-bca5-505fbf4c70bc-kube-api-access-m6rdk\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.552770 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.552688    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.654117 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.654016    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.654117 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.654093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c21653-ad82-4871-bca5-505fbf4c70bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.654350 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.654193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6rdk\" (UniqueName: \"kubernetes.io/projected/c7c21653-ad82-4871-bca5-505fbf4c70bc-kube-api-access-m6rdk\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.654350 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.654228    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.654507 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:00:45.654363    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-serving-cert: secret "maas-api-e2e-shared-a-w3-48898f-serving-cert" not found
Sep 11 15:00:45.654507 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:00:45.654433    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-maas-api-tls podName:c7c21653-ad82-4871-bca5-505fbf4c70bc nodeName:}" failed. No retries permitted until 2026-09-11 15:00:46.154412482 +0000 UTC m=+1252.477040832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-maas-api-tls") pod "maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2" (UID: "c7c21653-ad82-4871-bca5-505fbf4c70bc") : secret "maas-api-e2e-shared-a-w3-48898f-serving-cert" not found
Sep 11 15:00:45.654964 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.654945    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c21653-ad82-4871-bca5-505fbf4c70bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.656869 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.656848    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.663765 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.663742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6rdk\" (UniqueName: \"kubernetes.io/projected/c7c21653-ad82-4871-bca5-505fbf4c70bc-kube-api-access-m6rdk\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:45.811198 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.811162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:45.921148 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:45.921042    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:46.020583 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:46.020544    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:46.159679 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:46.159640    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:46.162125 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:46.162093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:46.303796 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:46.303760    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:46.576229 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:46.576204    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"]
Sep 11 15:00:46.577779 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:00:46.577756    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c21653_ad82_4871_bca5_505fbf4c70bc.slice/crio-c0ebf0d60d37e98d25cc883c5cd03011702127e75f216e42ec5e73a761db98e2 WatchSource:0}: Error finding container c0ebf0d60d37e98d25cc883c5cd03011702127e75f216e42ec5e73a761db98e2: Status 404 returned error can't find the container with id c0ebf0d60d37e98d25cc883c5cd03011702127e75f216e42ec5e73a761db98e2
Sep 11 15:00:47.414147 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:47.414098    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2" event={"ID":"c7c21653-ad82-4871-bca5-505fbf4c70bc","Type":"ContainerStarted","Data":"4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223"}
Sep 11 15:00:47.414333 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:47.414166    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2" event={"ID":"c7c21653-ad82-4871-bca5-505fbf4c70bc","Type":"ContainerStarted","Data":"c0ebf0d60d37e98d25cc883c5cd03011702127e75f216e42ec5e73a761db98e2"}
Sep 11 15:00:47.414333 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:47.414192    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:47.435694 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:47.435645    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2" podStartSLOduration=2.080782271 podStartE2EDuration="2.435628441s" totalImagesPullingTime="354.84617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:00:47.433926661 +0000 UTC m=+1253.756555030" watchObservedRunningTime="2026-09-11 15:00:47.435628441 +0000 UTC m=+1253.758256811"
Sep 11 15:00:53.424357 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:53.424325    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:00:56.712973 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:56.712935    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:00:58.327880 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:00:58.327846    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:01:00.106504 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:00.106471    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"]
Sep 11 15:01:00.106942 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:00.106676    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9" podUID="c9644e5d-0b72-475b-be30-72a4674e2d07" containerName="istio-proxy" containerID="cri-o://3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a" gracePeriod=30
Sep 11 15:01:00.136075 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:00.136045    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:00.354664 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:00.354626    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:00.482408 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:00.482342    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:05.367255 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.367223    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 15:01:05.447148 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447060    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-socket\") pod \"c9644e5d-0b72-475b-be30-72a4674e2d07\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") "
Sep 11 15:01:05.447148 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447104    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-data\") pod \"c9644e5d-0b72-475b-be30-72a4674e2d07\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") "
Sep 11 15:01:05.447404 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447151    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-token\") pod \"c9644e5d-0b72-475b-be30-72a4674e2d07\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") "
Sep 11 15:01:05.447404 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447169    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-envoy\") pod \"c9644e5d-0b72-475b-be30-72a4674e2d07\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") "
Sep 11 15:01:05.447404 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447215    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-certs\") pod \"c9644e5d-0b72-475b-be30-72a4674e2d07\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") "
Sep 11 15:01:05.447404 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447238    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dlbt\" (UniqueName: \"kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-kube-api-access-5dlbt\") pod \"c9644e5d-0b72-475b-be30-72a4674e2d07\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") "
Sep 11 15:01:05.447404 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447264    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-podinfo\") pod \"c9644e5d-0b72-475b-be30-72a4674e2d07\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") "
Sep 11 15:01:05.447404 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447306    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-credential-socket\") pod \"c9644e5d-0b72-475b-be30-72a4674e2d07\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") "
Sep 11 15:01:05.447404 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9644e5d-0b72-475b-be30-72a4674e2d07-istiod-ca-cert\") pod \"c9644e5d-0b72-475b-be30-72a4674e2d07\" (UID: \"c9644e5d-0b72-475b-be30-72a4674e2d07\") "
Sep 11 15:01:05.447816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447419    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c9644e5d-0b72-475b-be30-72a4674e2d07" (UID: "c9644e5d-0b72-475b-be30-72a4674e2d07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:05.447816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447477    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-data" (OuterVolumeSpecName: "istio-data") pod "c9644e5d-0b72-475b-be30-72a4674e2d07" (UID: "c9644e5d-0b72-475b-be30-72a4674e2d07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:05.447816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447666    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c9644e5d-0b72-475b-be30-72a4674e2d07" (UID: "c9644e5d-0b72-475b-be30-72a4674e2d07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:05.447816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447682    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c9644e5d-0b72-475b-be30-72a4674e2d07" (UID: "c9644e5d-0b72-475b-be30-72a4674e2d07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:05.447816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447792    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:05.447816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447812    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-data\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:05.448121 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447825    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-workload-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:05.448121 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447837    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-credential-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:05.448121 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.447970    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9644e5d-0b72-475b-be30-72a4674e2d07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c9644e5d-0b72-475b-be30-72a4674e2d07" (UID: "c9644e5d-0b72-475b-be30-72a4674e2d07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:01:05.449602 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.449573    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-token" (OuterVolumeSpecName: "istio-token") pod "c9644e5d-0b72-475b-be30-72a4674e2d07" (UID: "c9644e5d-0b72-475b-be30-72a4674e2d07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:01:05.449729 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.449604    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c9644e5d-0b72-475b-be30-72a4674e2d07" (UID: "c9644e5d-0b72-475b-be30-72a4674e2d07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:05.449729 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.449639    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c9644e5d-0b72-475b-be30-72a4674e2d07" (UID: "c9644e5d-0b72-475b-be30-72a4674e2d07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 15:01:05.449729 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.449653    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-kube-api-access-5dlbt" (OuterVolumeSpecName: "kube-api-access-5dlbt") pod "c9644e5d-0b72-475b-be30-72a4674e2d07" (UID: "c9644e5d-0b72-475b-be30-72a4674e2d07"). InnerVolumeSpecName "kube-api-access-5dlbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:01:05.454552 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.454528    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"]
Sep 11 15:01:05.506682 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.506645    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:05.512238 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.512208    2559 generic.go:356] "Generic (PLEG): container finished" podID="c9644e5d-0b72-475b-be30-72a4674e2d07" containerID="3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a" exitCode=0
Sep 11 15:01:05.512395 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.512249    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"
Sep 11 15:01:05.512395 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.512287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9" event={"ID":"c9644e5d-0b72-475b-be30-72a4674e2d07","Type":"ContainerDied","Data":"3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a"}
Sep 11 15:01:05.512395 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.512338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9" event={"ID":"c9644e5d-0b72-475b-be30-72a4674e2d07","Type":"ContainerDied","Data":"78ba9e0dc462c2cc6dae9afa3bbaf25ef12f8e5884f33c0ed537e1c2d1d0f932"}
Sep 11 15:01:05.512395 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.512363    2559 scope.go:117] "RemoveContainer" containerID="3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a"
Sep 11 15:01:05.521966 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.521948    2559 scope.go:117] "RemoveContainer" containerID="3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a"
Sep 11 15:01:05.522257 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:01:05.522238    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a\": container with ID starting with 3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a not found: ID does not exist" containerID="3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a"
Sep 11 15:01:05.522332 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.522266    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a"} err="failed to get container status \"3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a\": rpc error: code = NotFound desc = could not find container \"3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a\": container with ID starting with 3e802cfe7249edd214b72818202eaa2957989da62befb385366b772aa055021a not found: ID does not exist"
Sep 11 15:01:05.535347 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.535303    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"]
Sep 11 15:01:05.539773 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.539749    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-cd2cc1-openshift-default-5d8b876f7b-fxfv9"]
Sep 11 15:01:05.549130 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.549112    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-token\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:05.549181 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.549134    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-envoy\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:05.549181 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.549143    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dlbt\" (UniqueName: \"kubernetes.io/projected/c9644e5d-0b72-475b-be30-72a4674e2d07-kube-api-access-5dlbt\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:05.549181 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.549151    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9644e5d-0b72-475b-be30-72a4674e2d07-istio-podinfo\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:05.549181 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:05.549159    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9644e5d-0b72-475b-be30-72a4674e2d07-istiod-ca-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:06.180090 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:06.180047    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9644e5d-0b72-475b-be30-72a4674e2d07" path="/var/lib/kubelet/pods/c9644e5d-0b72-475b-be30-72a4674e2d07/volumes"
Sep 11 15:01:07.538750 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:07.538703    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:08.234747 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:08.234709    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:09.104989 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:09.104956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:09.804046 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:09.804010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:10.580406 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:10.580367    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:11.243106 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:11.243064    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:11.344679 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:11.344644    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:11.608448 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:11.608353    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:11.906363 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:11.906284    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"]
Sep 11 15:01:11.906908 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:11.906888    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9644e5d-0b72-475b-be30-72a4674e2d07" containerName="istio-proxy"
Sep 11 15:01:11.906908 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:11.906908    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9644e5d-0b72-475b-be30-72a4674e2d07" containerName="istio-proxy"
Sep 11 15:01:11.907053 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:11.906991    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9644e5d-0b72-475b-be30-72a4674e2d07" containerName="istio-proxy"
Sep 11 15:01:11.909492 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:11.909476    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:11.912275 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:11.912254    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-928943-openshift-default-dockercfg-42v6q\""
Sep 11 15:01:11.922605 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:11.922585    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"]
Sep 11 15:01:11.947986 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:11.947962    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:12.013236 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.013204    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-data\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.013402 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.013275    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssh8w\" (UniqueName: \"kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-kube-api-access-ssh8w\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.013402 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.013329    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-certs\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.013521 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.013422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-envoy\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.013521 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.013471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-socket\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.013521 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.013498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-token\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.013676 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.013524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b0414f2-22d7-48f6-996f-43e4bd222e68-istiod-ca-cert\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.013676 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.013658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-credential-socket\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.013776 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.013713    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-podinfo\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.042892 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.042860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:12.114506 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.114473    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-data\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.114632 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.114546    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssh8w\" (UniqueName: \"kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-kube-api-access-ssh8w\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.114632 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.114567    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-certs\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.114632 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.114605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-envoy\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.114632 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.114627    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-socket\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.114836 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.114652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-token\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.114836 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.114684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b0414f2-22d7-48f6-996f-43e4bd222e68-istiod-ca-cert\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.114935 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.114858    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-data\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.114935 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.114914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-certs\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.115040 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.115018    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-socket\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.115133 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.114787    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-credential-socket\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.115175 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.115139    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-credential-socket\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.115216 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.115185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-podinfo\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.115477 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.115447    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b0414f2-22d7-48f6-996f-43e4bd222e68-istiod-ca-cert\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.116820 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.116797    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-envoy\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.117386 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.117360    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-podinfo\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.123604 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.123582    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-token\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.123699 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.123664    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssh8w\" (UniqueName: \"kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-kube-api-access-ssh8w\") pod \"e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") " pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.222844 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.222778    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:12.290188 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.290143    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:12.506559 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.506534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"]
Sep 11 15:01:12.508165 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:01:12.508140    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0414f2_22d7_48f6_996f_43e4bd222e68.slice/crio-b01d24db2869205ca98037eb85fc2e27d547490da8bdae88a455a497501c04e0 WatchSource:0}: Error finding container b01d24db2869205ca98037eb85fc2e27d547490da8bdae88a455a497501c04e0: Status 404 returned error can't find the container with id b01d24db2869205ca98037eb85fc2e27d547490da8bdae88a455a497501c04e0
Sep 11 15:01:12.510585 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.510555    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:01:12.510667 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.510620    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:01:12.510667 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.510650    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:01:12.556803 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:12.556775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" event={"ID":"4b0414f2-22d7-48f6-996f-43e4bd222e68","Type":"ContainerStarted","Data":"b01d24db2869205ca98037eb85fc2e27d547490da8bdae88a455a497501c04e0"}
Sep 11 15:01:13.564238 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:13.564196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" event={"ID":"4b0414f2-22d7-48f6-996f-43e4bd222e68","Type":"ContainerStarted","Data":"9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948"}
Sep 11 15:01:13.587787 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:13.587736    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" podStartSLOduration=2.587721002 podStartE2EDuration="2.587721002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:01:13.585948786 +0000 UTC m=+1279.908577179" watchObservedRunningTime="2026-09-11 15:01:13.587721002 +0000 UTC m=+1279.910349371"
Sep 11 15:01:14.223263 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:14.223229    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:14.228605 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:14.228574    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:14.570377 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:14.570290    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:14.571485 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:14.571443    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:14.915356 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:14.915251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:15.657684 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:15.657646    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:15.763121 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:15.763086    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:15.874746 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:15.874713    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:16.781878 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:16.781847    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:17.561116 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:17.561075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:18.415867 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:18.415829    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:19.088416 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:19.088376    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:19.754339 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:19.754302    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:20.474989 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.474943    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q" podUID="b1e0e918-9125-4b5a-8d91-b811a6fd7172" containerName="main" containerID="cri-o://a58d672c0e9093863f3db8fef72b3308450ed964988a4742ef266b752287fd25" gracePeriod=45
Sep 11 15:01:20.510789 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.510758    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:20.601659 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.601629    2559 generic.go:356] "Generic (PLEG): container finished" podID="b1e0e918-9125-4b5a-8d91-b811a6fd7172" containerID="a58d672c0e9093863f3db8fef72b3308450ed964988a4742ef266b752287fd25" exitCode=0
Sep 11 15:01:20.601787 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.601711    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q" event={"ID":"b1e0e918-9125-4b5a-8d91-b811a6fd7172","Type":"ContainerDied","Data":"a58d672c0e9093863f3db8fef72b3308450ed964988a4742ef266b752287fd25"}
Sep 11 15:01:20.611723 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.611690    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:20.723670 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.723645    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:01:20.903701 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.903611    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-dshm\") pod \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") "
Sep 11 15:01:20.903701 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.903654    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-home\") pod \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") "
Sep 11 15:01:20.904215 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.903794    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpjpp\" (UniqueName: \"kubernetes.io/projected/b1e0e918-9125-4b5a-8d91-b811a6fd7172-kube-api-access-fpjpp\") pod \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") "
Sep 11 15:01:20.904215 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.903849    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tmp-dir\") pod \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") "
Sep 11 15:01:20.904215 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.903886    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tls-certs\") pod \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") "
Sep 11 15:01:20.904215 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.903912    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-model-cache\") pod \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\" (UID: \"b1e0e918-9125-4b5a-8d91-b811a6fd7172\") "
Sep 11 15:01:20.904215 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.903923    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-home" (OuterVolumeSpecName: "home") pod "b1e0e918-9125-4b5a-8d91-b811a6fd7172" (UID: "b1e0e918-9125-4b5a-8d91-b811a6fd7172"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:20.904215 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.904075    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1e0e918-9125-4b5a-8d91-b811a6fd7172" (UID: "b1e0e918-9125-4b5a-8d91-b811a6fd7172"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:20.904215 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.904192    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1e0e918-9125-4b5a-8d91-b811a6fd7172" (UID: "b1e0e918-9125-4b5a-8d91-b811a6fd7172"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:20.904641 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.904305    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:20.904641 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.904325    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:20.904641 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.904335    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:20.906019 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.905998    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e0e918-9125-4b5a-8d91-b811a6fd7172-kube-api-access-fpjpp" (OuterVolumeSpecName: "kube-api-access-fpjpp") pod "b1e0e918-9125-4b5a-8d91-b811a6fd7172" (UID: "b1e0e918-9125-4b5a-8d91-b811a6fd7172"). InnerVolumeSpecName "kube-api-access-fpjpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:01:20.906147 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.906093    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-dshm" (OuterVolumeSpecName: "dshm") pod "b1e0e918-9125-4b5a-8d91-b811a6fd7172" (UID: "b1e0e918-9125-4b5a-8d91-b811a6fd7172"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:20.906147 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.906119    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1e0e918-9125-4b5a-8d91-b811a6fd7172" (UID: "b1e0e918-9125-4b5a-8d91-b811a6fd7172"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:01:20.937433 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:20.937398    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:21.005572 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:21.005547    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e0e918-9125-4b5a-8d91-b811a6fd7172-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:21.005572 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:21.005569    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpjpp\" (UniqueName: \"kubernetes.io/projected/b1e0e918-9125-4b5a-8d91-b811a6fd7172-kube-api-access-fpjpp\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:21.005745 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:21.005578    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e0e918-9125-4b5a-8d91-b811a6fd7172-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:01:21.119142 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:21.119107    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:21.218214 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:21.218121    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:21.608093 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:21.608062    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"
Sep 11 15:01:21.608264 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:21.608099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q" event={"ID":"b1e0e918-9125-4b5a-8d91-b811a6fd7172","Type":"ContainerDied","Data":"2508e9e90473b96fd716380505d7d4513539d9c6909f583fe10dac3be069a37b"}
Sep 11 15:01:21.608264 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:21.608149    2559 scope.go:117] "RemoveContainer" containerID="a58d672c0e9093863f3db8fef72b3308450ed964988a4742ef266b752287fd25"
Sep 11 15:01:21.629789 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:21.629764    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"]
Sep 11 15:01:21.635534 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:21.635511    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-cd2cc1/noten-model-cd2cc1-kserve-8d578c7d7-wgl8q"]
Sep 11 15:01:22.180359 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:22.180326    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e0e918-9125-4b5a-8d91-b811a6fd7172" path="/var/lib/kubelet/pods/b1e0e918-9125-4b5a-8d91-b811a6fd7172/volumes"
Sep 11 15:01:26.563091 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.563058    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"]
Sep 11 15:01:26.563607 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.563541    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e0e918-9125-4b5a-8d91-b811a6fd7172" containerName="main"
Sep 11 15:01:26.563607 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.563558    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e0e918-9125-4b5a-8d91-b811a6fd7172" containerName="main"
Sep 11 15:01:26.563699 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.563635    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e0e918-9125-4b5a-8d91-b811a6fd7172" containerName="main"
Sep 11 15:01:26.565620 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.565603    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.568557 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.568528    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"distinct-2-w0-e7ca56-kserve-self-signed-certs\""
Sep 11 15:01:26.577652 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.577615    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"]
Sep 11 15:01:26.660634 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.660603    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-home\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.660634 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.660642    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-model-cache\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.660837 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.660769    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-tmp-dir\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.660837 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.660806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-dshm\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.660837 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.660827    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46b1110-8118-457d-a50c-fad9d9aee29a-tls-certs\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.660996 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.660914    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mljhd\" (UniqueName: \"kubernetes.io/projected/c46b1110-8118-457d-a50c-fad9d9aee29a-kube-api-access-mljhd\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.761999 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.761971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-home\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.762157 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.762004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-model-cache\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.762157 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.762061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-tmp-dir\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.762157 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.762086    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-dshm\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.762157 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.762101    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46b1110-8118-457d-a50c-fad9d9aee29a-tls-certs\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.762157 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.762122    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mljhd\" (UniqueName: \"kubernetes.io/projected/c46b1110-8118-457d-a50c-fad9d9aee29a-kube-api-access-mljhd\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.762492 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.762471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-home\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.762556 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.762509    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-tmp-dir\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.762556 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.762531    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-model-cache\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.764407 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.764388    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-dshm\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.764611 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.764593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46b1110-8118-457d-a50c-fad9d9aee29a-tls-certs\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.771134 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.771114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mljhd\" (UniqueName: \"kubernetes.io/projected/c46b1110-8118-457d-a50c-fad9d9aee29a-kube-api-access-mljhd\") pod \"distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.875741 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.875654    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:26.969316 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:26.969267    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:27.147217 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:27.147137    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"]
Sep 11 15:01:27.150425 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:01:27.150392    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc46b1110_8118_457d_a50c_fad9d9aee29a.slice/crio-ed5d625ba65bc278154ccb7d81fbef45de56f83432eeab4ffab59a8e4549890d WatchSource:0}: Error finding container ed5d625ba65bc278154ccb7d81fbef45de56f83432eeab4ffab59a8e4549890d: Status 404 returned error can't find the container with id ed5d625ba65bc278154ccb7d81fbef45de56f83432eeab4ffab59a8e4549890d
Sep 11 15:01:27.642306 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:27.642268    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2" event={"ID":"c46b1110-8118-457d-a50c-fad9d9aee29a","Type":"ContainerStarted","Data":"82a7e867d0e1a5ef2e80a2b840d3ae9c6b8edd2c603b85d841cf0ed89a1b42ea"}
Sep 11 15:01:27.642712 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:27.642318    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2" event={"ID":"c46b1110-8118-457d-a50c-fad9d9aee29a","Type":"ContainerStarted","Data":"ed5d625ba65bc278154ccb7d81fbef45de56f83432eeab4ffab59a8e4549890d"}
Sep 11 15:01:27.660276 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:27.660225    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2" podStartSLOduration=1.6602096259999999 podStartE2EDuration="1.660209626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:01:27.658549246 +0000 UTC m=+1293.981177616" watchObservedRunningTime="2026-09-11 15:01:27.660209626 +0000 UTC m=+1293.982838036"
Sep 11 15:01:33.160921 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:33.160881    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:34.019279 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:34.019237    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:36.431986 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:36.431928    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:01:36.876693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:36.876654    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:36.876900 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:36.876711    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:36.894134 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:36.894104    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:37.709340 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:37.709303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:01:38.872346 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.872304    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"]
Sep 11 15:01:38.889145 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.889118    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"]
Sep 11 15:01:38.889479 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.889411    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:38.892888 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.892867    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a843ae03\"/\"kube-root-ca.crt\""
Sep 11 15:01:38.898165 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.897848    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a843ae03\"/\"default-dockercfg-q6t92\""
Sep 11 15:01:38.898165 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.898122    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a843ae03\"/\"e2e-lifecycle-model-a843ae03-kserve-self-signed-certs\""
Sep 11 15:01:38.898349 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.898323    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a843ae03\"/\"openshift-service-ca.crt\""
Sep 11 15:01:38.901490 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.901403    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:38.986777 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.986748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkcpw\" (UniqueName: \"kubernetes.io/projected/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-kube-api-access-zkcpw\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:38.986971 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.986784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tls-certs\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:38.986971 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.986860    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-model-cache\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:38.986971 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.986878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-home\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:38.987145 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.987008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-dshm\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:38.987145 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:38.987064    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tmp-dir\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.088283 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.088242    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-model-cache\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.088283 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.088286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-home\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.088554 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.088344    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-dshm\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.088554 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.088376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tmp-dir\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.088554 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.088407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkcpw\" (UniqueName: \"kubernetes.io/projected/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-kube-api-access-zkcpw\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.088658 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.088584    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tls-certs\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.088755 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.088736    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-model-cache\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.088821 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.088800    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-home\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.089002 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.088981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tmp-dir\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.090651 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.090632    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-dshm\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.091084 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.091062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tls-certs\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.099495 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.099471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkcpw\" (UniqueName: \"kubernetes.io/projected/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-kube-api-access-zkcpw\") pod \"e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") " pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.206254 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.206174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:39.492299 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.492272    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"]
Sep 11 15:01:39.494378 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:01:39.494348    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb63229d6_1b4c_4e9a_97f9_6d20ccd486e7.slice/crio-974f7055263bbf8b422db3b4c4faa35de01632592c4d47f15536c4a1f5a00065 WatchSource:0}: Error finding container 974f7055263bbf8b422db3b4c4faa35de01632592c4d47f15536c4a1f5a00065: Status 404 returned error can't find the container with id 974f7055263bbf8b422db3b4c4faa35de01632592c4d47f15536c4a1f5a00065
Sep 11 15:01:39.709367 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.709329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc" event={"ID":"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7","Type":"ContainerStarted","Data":"3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2"}
Sep 11 15:01:39.709553 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.709375    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc" event={"ID":"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7","Type":"ContainerStarted","Data":"974f7055263bbf8b422db3b4c4faa35de01632592c4d47f15536c4a1f5a00065"}
Sep 11 15:01:39.732125 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:39.732070    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc" podStartSLOduration=1.7320520130000001 podStartE2EDuration="1.732052013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:01:39.730071416 +0000 UTC m=+1306.052699786" watchObservedRunningTime="2026-09-11 15:01:39.732052013 +0000 UTC m=+1306.054680413"
Sep 11 15:01:42.702670 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:42.702623    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:43.086816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:43.086783    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:43.615963 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:43.615915    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:44.177665 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:44.177625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:44.428437 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:44.428346    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:45.306178 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:45.306141    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:45.611689 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:45.611603    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 15:01:45.611689 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:45.611680    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 15:01:45.714433 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:45.714400    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:46.129951 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:46.129929    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:46.751331 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:46.751273    2559 generic.go:356] "Generic (PLEG): container finished" podID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerID="9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948" exitCode=0
Sep 11 15:01:46.751777 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:46.751346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" event={"ID":"4b0414f2-22d7-48f6-996f-43e4bd222e68","Type":"ContainerDied","Data":"9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948"}
Sep 11 15:01:46.751834 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:46.751797    2559 scope.go:117] "RemoveContainer" containerID="9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948"
Sep 11 15:01:46.753068 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:46.753044    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:01:46.753148 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:46.753111    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:01:46.753148 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:46.753138    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:01:46.857960 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:46.857928    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:47.563630 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:47.563579    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:47.760502 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:47.760441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" event={"ID":"4b0414f2-22d7-48f6-996f-43e4bd222e68","Type":"ContainerStarted","Data":"ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600"}
Sep 11 15:01:48.223109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:48.223073    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:48.223329 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:48.223237    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 15:01:48.223329 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:48.223291    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 15:01:48.335666 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:48.335629    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:49.206591 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:49.206550    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:49.207049 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:49.206828    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:49.219285 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:49.219254    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:49.223372 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:49.223350    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 15:01:49.223469 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:49.223402    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 15:01:49.427240 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:49.427200    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:49.506564 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:49.506532    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:49.794022 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:49.793943    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:01:50.224099 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:50.224071    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 15:01:50.224508 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:50.224121    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 15:01:50.614549 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:50.614449    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:01:51.223566 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:51.223531    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 15:01:51.223756 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:51.223589    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 15:01:52.223883 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:52.223845    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 15:01:52.224376 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:52.223908    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 15:01:53.226776 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:53.226743    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:53.227198 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:53.227178    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:01:53.227888 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:01:53.227866    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:02:09.416580 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:09.416544    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:02:16.085939 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:16.085902    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:20.602841 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:20.602803    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:25.595865 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:25.595825    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:26.674882 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:26.674852    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:27.226883 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:27.226847    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:27.998030 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:27.997997    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:28.814068 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:28.814025    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:29.703947 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:29.703908    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:30.414006 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:30.413964    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:31.112829 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:31.112786    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:31.802192 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:31.802155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:36.626859 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:36.626823    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:02:41.018686 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.018649    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"]
Sep 11 15:02:41.019183 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.018894    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz" podUID="fbda0a30-75bb-475d-aacc-80bf75dafbce" containerName="maas-api" containerID="cri-o://ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22" gracePeriod=30
Sep 11 15:02:41.045878 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.045850    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:41.115726 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.115686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:41.184221 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.184188    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:41.254854 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.254816    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:41.270202 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.270144    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 15:02:41.315118 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.315090    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-maas-api-tls\") pod \"fbda0a30-75bb-475d-aacc-80bf75dafbce\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") "
Sep 11 15:02:41.315314 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.315166    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc6cw\" (UniqueName: \"kubernetes.io/projected/fbda0a30-75bb-475d-aacc-80bf75dafbce-kube-api-access-tc6cw\") pod \"fbda0a30-75bb-475d-aacc-80bf75dafbce\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") "
Sep 11 15:02:41.315314 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.315240    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-metrics-certs\") pod \"fbda0a30-75bb-475d-aacc-80bf75dafbce\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") "
Sep 11 15:02:41.315314 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.315309    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbda0a30-75bb-475d-aacc-80bf75dafbce-openshift-service-ca\") pod \"fbda0a30-75bb-475d-aacc-80bf75dafbce\" (UID: \"fbda0a30-75bb-475d-aacc-80bf75dafbce\") "
Sep 11 15:02:41.315729 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.315702    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbda0a30-75bb-475d-aacc-80bf75dafbce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fbda0a30-75bb-475d-aacc-80bf75dafbce" (UID: "fbda0a30-75bb-475d-aacc-80bf75dafbce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:02:41.317217 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.317186    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fbda0a30-75bb-475d-aacc-80bf75dafbce" (UID: "fbda0a30-75bb-475d-aacc-80bf75dafbce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:02:41.317707 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.317679    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fbda0a30-75bb-475d-aacc-80bf75dafbce" (UID: "fbda0a30-75bb-475d-aacc-80bf75dafbce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:02:41.317707 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.317696    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbda0a30-75bb-475d-aacc-80bf75dafbce-kube-api-access-tc6cw" (OuterVolumeSpecName: "kube-api-access-tc6cw") pod "fbda0a30-75bb-475d-aacc-80bf75dafbce" (UID: "fbda0a30-75bb-475d-aacc-80bf75dafbce"). InnerVolumeSpecName "kube-api-access-tc6cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:02:41.322300 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.322269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:41.417148 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.417112    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:02:41.417148 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.417148    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbda0a30-75bb-475d-aacc-80bf75dafbce-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:02:41.417148 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.417158    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbda0a30-75bb-475d-aacc-80bf75dafbce-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:02:41.417496 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.417167    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc6cw\" (UniqueName: \"kubernetes.io/projected/fbda0a30-75bb-475d-aacc-80bf75dafbce-kube-api-access-tc6cw\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:02:41.696163 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:41.696072    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:42.070830 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:42.070798    2559 generic.go:356] "Generic (PLEG): container finished" podID="fbda0a30-75bb-475d-aacc-80bf75dafbce" containerID="ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22" exitCode=0
Sep 11 15:02:42.071355 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:42.070840    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"
Sep 11 15:02:42.071355 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:42.070869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz" event={"ID":"fbda0a30-75bb-475d-aacc-80bf75dafbce","Type":"ContainerDied","Data":"ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22"}
Sep 11 15:02:42.071355 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:42.070911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz" event={"ID":"fbda0a30-75bb-475d-aacc-80bf75dafbce","Type":"ContainerDied","Data":"b2c0826ddfd4f4f4b492ce499311d928171b4122149d86306a79c35e20559dcf"}
Sep 11 15:02:42.071355 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:42.070932    2559 scope.go:117] "RemoveContainer" containerID="ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22"
Sep 11 15:02:42.080413 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:42.080396    2559 scope.go:117] "RemoveContainer" containerID="ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22"
Sep 11 15:02:42.080745 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:02:42.080726    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22\": container with ID starting with ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22 not found: ID does not exist" containerID="ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22"
Sep 11 15:02:42.080799 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:42.080755    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22"} err="failed to get container status \"ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22\": rpc error: code = NotFound desc = could not find container \"ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22\": container with ID starting with ae36cdf5729625db731940a6bc84c1cfcf6b7e243349141dc47a83fdbd43eb22 not found: ID does not exist"
Sep 11 15:02:42.095005 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:42.094982    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"]
Sep 11 15:02:42.096938 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:42.096919    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a843ae03-77585f9d75-ldzsz"]
Sep 11 15:02:42.180199 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:42.180168    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbda0a30-75bb-475d-aacc-80bf75dafbce" path="/var/lib/kubelet/pods/fbda0a30-75bb-475d-aacc-80bf75dafbce/volumes"
Sep 11 15:02:42.606210 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:42.606171    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:43.369747 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:43.369713    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:44.213949 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:44.213920    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:44.539444 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:44.539412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:44.913515 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:44.913396    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:45.713507 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:45.713442    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:46.402694 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:46.402657    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:02:49.521012 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:49.520978    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"]
Sep 11 15:02:49.571009 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:02:49.570970    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:01.214774 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:01.214736    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:04.540184 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.540144    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc" podUID="b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" containerName="main" containerID="cri-o://3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2" gracePeriod=45
Sep 11 15:03:04.800909 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.800845    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:03:04.953660 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.953629    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tls-certs\") pod \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") "
Sep 11 15:03:04.953819 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.953685    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-home\") pod \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") "
Sep 11 15:03:04.953819 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.953761    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkcpw\" (UniqueName: \"kubernetes.io/projected/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-kube-api-access-zkcpw\") pod \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") "
Sep 11 15:03:04.953819 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.953785    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-model-cache\") pod \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") "
Sep 11 15:03:04.953819 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.953809    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tmp-dir\") pod \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") "
Sep 11 15:03:04.953984 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.953834    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-dshm\") pod \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\" (UID: \"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7\") "
Sep 11 15:03:04.953984 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.953941    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-home" (OuterVolumeSpecName: "home") pod "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" (UID: "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:03:04.954106 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.954080    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" (UID: "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:03:04.954163 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.954089    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" (UID: "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:03:04.954220 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.954188    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:03:04.954220 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.954207    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:03:04.954293 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.954236    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:03:04.955901 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.955869    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-dshm" (OuterVolumeSpecName: "dshm") pod "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" (UID: "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:03:04.956039 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.955913    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" (UID: "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:03:04.956039 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:04.955932    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-kube-api-access-zkcpw" (OuterVolumeSpecName: "kube-api-access-zkcpw") pod "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" (UID: "b63229d6-1b4c-4e9a-97f9-6d20ccd486e7"). InnerVolumeSpecName "kube-api-access-zkcpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:03:05.055634 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.055540    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkcpw\" (UniqueName: \"kubernetes.io/projected/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-kube-api-access-zkcpw\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:03:05.055634 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.055569    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:03:05.055634 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.055579    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:03:05.069915 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.069882    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"]
Sep 11 15:03:05.070309 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.070294    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbda0a30-75bb-475d-aacc-80bf75dafbce" containerName="maas-api"
Sep 11 15:03:05.070309 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.070309    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbda0a30-75bb-475d-aacc-80bf75dafbce" containerName="maas-api"
Sep 11 15:03:05.070491 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.070330    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" containerName="main"
Sep 11 15:03:05.070491 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.070336    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" containerName="main"
Sep 11 15:03:05.070491 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.070397    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbda0a30-75bb-475d-aacc-80bf75dafbce" containerName="maas-api"
Sep 11 15:03:05.070491 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.070408    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" containerName="main"
Sep 11 15:03:05.073785 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.073766    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.076720 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.076698    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"distinct-2-w4-837632-kserve-self-signed-certs\""
Sep 11 15:03:05.083939 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.083919    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"]
Sep 11 15:03:05.106982 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.106956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:05.198157 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.198127    2559 generic.go:356] "Generic (PLEG): container finished" podID="b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" containerID="3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2" exitCode=0
Sep 11 15:03:05.198319 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.198168    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"
Sep 11 15:03:05.198319 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.198218    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc" event={"ID":"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7","Type":"ContainerDied","Data":"3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2"}
Sep 11 15:03:05.198319 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.198261    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc" event={"ID":"b63229d6-1b4c-4e9a-97f9-6d20ccd486e7","Type":"ContainerDied","Data":"974f7055263bbf8b422db3b4c4faa35de01632592c4d47f15536c4a1f5a00065"}
Sep 11 15:03:05.198319 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.198284    2559 scope.go:117] "RemoveContainer" containerID="3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2"
Sep 11 15:03:05.207663 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.207646    2559 scope.go:117] "RemoveContainer" containerID="3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2"
Sep 11 15:03:05.207926 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:03:05.207906    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2\": container with ID starting with 3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2 not found: ID does not exist" containerID="3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2"
Sep 11 15:03:05.207976 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.207935    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2"} err="failed to get container status \"3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2\": rpc error: code = NotFound desc = could not find container \"3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2\": container with ID starting with 3ca86b269431184bcd2e4e5929d59d8b0818df1e21e2dac2cf0614099a6695e2 not found: ID does not exist"
Sep 11 15:03:05.220744 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.220724    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"]
Sep 11 15:03:05.223092 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.223072    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a843ae03/e2e-lifecycle-model-a843ae03-kserve-66ddc6b967-4k9pc"]
Sep 11 15:03:05.257602 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.257569    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tmp-dir\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.257775 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.257616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tls-certs\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.257775 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.257647    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-dshm\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.257775 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.257673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42shw\" (UniqueName: \"kubernetes.io/projected/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-kube-api-access-42shw\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.257775 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.257753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-model-cache\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.257990 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.257779    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-home\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.359177 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.359075    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tmp-dir\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.359177 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.359139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tls-certs\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.359177 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.359170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-dshm\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.359482 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.359197    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42shw\" (UniqueName: \"kubernetes.io/projected/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-kube-api-access-42shw\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.359482 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.359264    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-model-cache\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.359482 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.359292    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-home\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.359655 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.359547    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tmp-dir\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.359655 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.359647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-home\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.359852 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.359830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-model-cache\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.361467 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.361431    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-dshm\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.361756 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.361737    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tls-certs\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.367703 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.367682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42shw\" (UniqueName: \"kubernetes.io/projected/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-kube-api-access-42shw\") pod \"distinct-2-w4-837632-kserve-68764589fc-r9pk2\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") " pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.383692 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.383662    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:05.656626 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:05.656591    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"]
Sep 11 15:03:05.658681 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:03:05.658651    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90ab43ad_10fc_4be9_9f30_e1d486dc93ea.slice/crio-8fd9c6e7b9555363735296c8d32cf9111870b7b1876249aa673d33ee2fcb2873 WatchSource:0}: Error finding container 8fd9c6e7b9555363735296c8d32cf9111870b7b1876249aa673d33ee2fcb2873: Status 404 returned error can't find the container with id 8fd9c6e7b9555363735296c8d32cf9111870b7b1876249aa673d33ee2fcb2873
Sep 11 15:03:06.180466 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:06.180432    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b63229d6-1b4c-4e9a-97f9-6d20ccd486e7" path="/var/lib/kubelet/pods/b63229d6-1b4c-4e9a-97f9-6d20ccd486e7/volumes"
Sep 11 15:03:06.203908 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:06.203872    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2" event={"ID":"90ab43ad-10fc-4be9-9f30-e1d486dc93ea","Type":"ContainerStarted","Data":"1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e"}
Sep 11 15:03:06.204064 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:06.203915    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2" event={"ID":"90ab43ad-10fc-4be9-9f30-e1d486dc93ea","Type":"ContainerStarted","Data":"8fd9c6e7b9555363735296c8d32cf9111870b7b1876249aa673d33ee2fcb2873"}
Sep 11 15:03:06.223691 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:06.223648    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2" podStartSLOduration=1.2236344909999999 podStartE2EDuration="1.223634491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:03:06.221788759 +0000 UTC m=+1392.544417131" watchObservedRunningTime="2026-09-11 15:03:06.223634491 +0000 UTC m=+1392.546262860"
Sep 11 15:03:09.217829 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:09.217791    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:03:15.384319 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:15.384277    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:15.384319 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:15.384321    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:15.396627 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:15.396604    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:16.266895 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:16.266868    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:03:22.768629 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.768595    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"]
Sep 11 15:03:22.774270 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.774245    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.777397 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.777375    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"kube-root-ca.crt\""
Sep 11 15:03:22.778622 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.778599    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"openshift-service-ca.crt\""
Sep 11 15:03:22.778731 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.778626    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"default-dockercfg-t992d\""
Sep 11 15:03:22.778731 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.778604    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"auth-test-model-928943-kserve-self-signed-certs\""
Sep 11 15:03:22.783032 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.783006    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"]
Sep 11 15:03:22.803843 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.803814    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:22.831002 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.830974    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-home\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.831118 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.831031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tmp-dir\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.831118 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.831053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tls-certs\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.831118 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.831099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwdx9\" (UniqueName: \"kubernetes.io/projected/ab9ca370-a41f-418d-9b7e-3800b6349cc8-kube-api-access-hwdx9\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.831261 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.831159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-model-cache\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.831261 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.831189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-dshm\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.931808 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.931775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-model-cache\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.932046 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.931818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-dshm\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.932046 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.931859    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-home\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.932046 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.931906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tmp-dir\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.932046 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.931931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tls-certs\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.932046 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.931976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwdx9\" (UniqueName: \"kubernetes.io/projected/ab9ca370-a41f-418d-9b7e-3800b6349cc8-kube-api-access-hwdx9\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.932321 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.932181    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-model-cache\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.932321 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.932239    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-home\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.932412 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.932331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tmp-dir\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.934104 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.934075    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-dshm\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.934478 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.934441    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tls-certs\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:22.940596 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:22.940575    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwdx9\" (UniqueName: \"kubernetes.io/projected/ab9ca370-a41f-418d-9b7e-3800b6349cc8-kube-api-access-hwdx9\") pod \"auth-test-model-928943-kserve-97946c58c-2sffs\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") " pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:23.085564 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:23.085444    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:23.358992 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:23.358957    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"]
Sep 11 15:03:23.360196 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:03:23.360166    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab9ca370_a41f_418d_9b7e_3800b6349cc8.slice/crio-02f95bb4070b42b7d974a62ba12a76215fbc0b6b8060a08e19152de42557e4d5 WatchSource:0}: Error finding container 02f95bb4070b42b7d974a62ba12a76215fbc0b6b8060a08e19152de42557e4d5: Status 404 returned error can't find the container with id 02f95bb4070b42b7d974a62ba12a76215fbc0b6b8060a08e19152de42557e4d5
Sep 11 15:03:24.298507 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:24.298472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs" event={"ID":"ab9ca370-a41f-418d-9b7e-3800b6349cc8","Type":"ContainerStarted","Data":"b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a"}
Sep 11 15:03:24.298507 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:24.298514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs" event={"ID":"ab9ca370-a41f-418d-9b7e-3800b6349cc8","Type":"ContainerStarted","Data":"02f95bb4070b42b7d974a62ba12a76215fbc0b6b8060a08e19152de42557e4d5"}
Sep 11 15:03:24.324059 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:24.323996    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs" podStartSLOduration=2.323980534 podStartE2EDuration="2.323980534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:03:24.321751914 +0000 UTC m=+1410.644380283" watchObservedRunningTime="2026-09-11 15:03:24.323980534 +0000 UTC m=+1410.646608901"
Sep 11 15:03:26.457199 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:26.457162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:28.564266 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:28.564226    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:33.086050 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:33.086015    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:33.086501 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:33.086063    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:33.098370 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:33.098343    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:33.355708 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:33.355623    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:03:34.885846 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:34.885812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:39.323127 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:39.323088    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:03:44.571336 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:44.571279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:45.506465 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:45.506420    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:48.025337 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:48.025249    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:59.366250 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:59.366202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:59.426106 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:59.426075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:03:59.657826 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:03:59.657740    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:00.965058 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:00.965024    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:01.055577 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:01.055544    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:01.156235 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:01.156180    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:01.870436 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:01.870392    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:01.967337 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:01.967284    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:02.064861 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:02.064824    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:03.371432 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:03.371389    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:09.362968 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:09.362927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:09.468994 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:09.468946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:09.563004 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:09.562963    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:10.099192 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:10.099158    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:13.221287 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:13.221250    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:04:16.335652 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:16.335610    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:16.407849 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:16.407806    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:16.487636 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:16.487596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:16.588535 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:16.588415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:27.087869 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:27.087831    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:27.332705 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:27.332662    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:35.561644 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:35.561596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:48.317050 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:48.317010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:54.017577 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:54.017537    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:54.989607 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:54.989572    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:55.073266 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:55.073222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:55.327995 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:55.327903    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:55.664867 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:55.664773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:55.921827 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:55.921730    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:57.021801 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:57.021764    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:04:58.686783 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:58.686743    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:59.310992 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:59.310958    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:59.417881 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:59.417842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:04:59.514664 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:04:59.514627    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:00.312747 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:00.312711    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:01.221638 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:01.221598    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:01.917559 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:01.917514    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:02.903975 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:02.903941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:03.612807 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:03.612765    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:04.318653 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.318623    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"]
Sep 11 15:05:04.322551 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.322527    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.325999 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.325978    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-54b7f9-serving-cert\""
Sep 11 15:05:04.339506 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.339481    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"]
Sep 11 15:05:04.380174 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.380139    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:04.398386 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.398355    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-maas-api-tls\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.398560 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.398425    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rsgl\" (UniqueName: \"kubernetes.io/projected/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-kube-api-access-2rsgl\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.398560 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.398540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-metrics-certs\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.398662 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.398589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-openshift-service-ca\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.493227 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.493187    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:04.499630 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.499606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-metrics-certs\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.499737 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.499666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-openshift-service-ca\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.499737 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.499729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-maas-api-tls\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.499851 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.499790    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rsgl\" (UniqueName: \"kubernetes.io/projected/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-kube-api-access-2rsgl\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.499914 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:05:04.499896    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-serving-cert: secret "maas-api-e2e-api-a-54b7f9-serving-cert" not found
Sep 11 15:05:04.499980 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:05:04.499969    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-maas-api-tls podName:2ed86dcf-37ce-4e4a-9d2e-c24a233f961b nodeName:}" failed. No retries permitted until 2026-09-11 15:05:04.999953732 +0000 UTC m=+1511.322582083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-maas-api-tls") pod "maas-api-e2e-api-a-54b7f9-559dc5988-bt82s" (UID: "2ed86dcf-37ce-4e4a-9d2e-c24a233f961b") : secret "maas-api-e2e-api-a-54b7f9-serving-cert" not found
Sep 11 15:05:04.500504 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.500484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-openshift-service-ca\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.502078 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.502060    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-metrics-certs\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.509644 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.509617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rsgl\" (UniqueName: \"kubernetes.io/projected/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-kube-api-access-2rsgl\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:04.594521 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.594405    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:04.916534 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:04.916417    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:05.005355 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:05.005311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-maas-api-tls\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:05.007667 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:05.007645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-maas-api-tls\") pod \"maas-api-e2e-api-a-54b7f9-559dc5988-bt82s\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:05.014477 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:05.014412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:05.232783 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:05.232684    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:05.509363 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:05.509338    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"]
Sep 11 15:05:05.511487 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:05:05.511436    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed86dcf_37ce_4e4a_9d2e_c24a233f961b.slice/crio-1ffcd95de85de5a38ead5207336e66e88ffe6a06384b486044a75476fcd484d8 WatchSource:0}: Error finding container 1ffcd95de85de5a38ead5207336e66e88ffe6a06384b486044a75476fcd484d8: Status 404 returned error can't find the container with id 1ffcd95de85de5a38ead5207336e66e88ffe6a06384b486044a75476fcd484d8
Sep 11 15:05:05.512895 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:05.512875    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 15:05:05.836529 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:05.836407    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s" event={"ID":"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b","Type":"ContainerStarted","Data":"1ffcd95de85de5a38ead5207336e66e88ffe6a06384b486044a75476fcd484d8"}
Sep 11 15:05:06.826255 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:06.826207    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:06.843938 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:06.843901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s" event={"ID":"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b","Type":"ContainerStarted","Data":"76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d"}
Sep 11 15:05:06.844139 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:06.844001    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:06.866060 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:06.865998    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s" podStartSLOduration=2.375424412 podStartE2EDuration="2.865980707s" totalImagesPullingTime="490.556295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:05:06.864087572 +0000 UTC m=+1513.186715944" watchObservedRunningTime="2026-09-11 15:05:06.865980707 +0000 UTC m=+1513.188609075"
Sep 11 15:05:07.301190 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:07.301154    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:08.016798 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:08.016762    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:08.717297 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:08.717255    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:09.523085 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:09.523032    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:11.107892 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:11.107860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:11.214609 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:11.214571    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:11.811966 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:11.811926    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"]
Sep 11 15:05:11.833372 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:11.833335    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"]
Sep 11 15:05:11.850619 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:11.850576    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"]
Sep 11 15:05:11.905998 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:11.905967    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:12.007732 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:12.007700    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:12.669994 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:12.669954    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:12.855210 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:12.855183    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:05:13.405097 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:13.405057    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:14.306685 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:14.306641    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:17.332800 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:17.332719    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:23.129047 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:23.129010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:23.213754 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:23.213717    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:23.314192 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:23.314160    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:23.414519 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:23.414402    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:23.730281 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:23.730252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:23.831094 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:23.831055    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:24.628101 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:24.628053    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:25.688281 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:25.688237    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:26.453441 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:26.453405    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:26.832103 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:26.832042    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2" podUID="c46b1110-8118-457d-a50c-fad9d9aee29a" containerName="main" containerID="cri-o://82a7e867d0e1a5ef2e80a2b840d3ae9c6b8edd2c603b85d841cf0ed89a1b42ea" gracePeriod=45
Sep 11 15:05:26.853667 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:26.853635    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf" podUID="b5b2b453-4c05-4a53-bedf-319e648831fd" containerName="main" containerID="cri-o://3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4" gracePeriod=45
Sep 11 15:05:26.872513 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:26.872446    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr" podUID="1f02f1ac-5778-4554-bfe0-8d03a17480dc" containerName="main" containerID="cri-o://abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846" gracePeriod=45
Sep 11 15:05:26.955620 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:26.955583    2559 generic.go:356] "Generic (PLEG): container finished" podID="c46b1110-8118-457d-a50c-fad9d9aee29a" containerID="82a7e867d0e1a5ef2e80a2b840d3ae9c6b8edd2c603b85d841cf0ed89a1b42ea" exitCode=0
Sep 11 15:05:26.955786 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:26.955663    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2" event={"ID":"c46b1110-8118-457d-a50c-fad9d9aee29a","Type":"ContainerDied","Data":"82a7e867d0e1a5ef2e80a2b840d3ae9c6b8edd2c603b85d841cf0ed89a1b42ea"}
Sep 11 15:05:27.133586 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.133537    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 15:05:27.146109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.145400    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-model-cache\") pod \"b5b2b453-4c05-4a53-bedf-319e648831fd\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") "
Sep 11 15:05:27.146109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.145476    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-home\") pod \"b5b2b453-4c05-4a53-bedf-319e648831fd\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") "
Sep 11 15:05:27.146109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.145547    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-tmp-dir\") pod \"b5b2b453-4c05-4a53-bedf-319e648831fd\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") "
Sep 11 15:05:27.146109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.145583    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzfc6\" (UniqueName: \"kubernetes.io/projected/b5b2b453-4c05-4a53-bedf-319e648831fd-kube-api-access-pzfc6\") pod \"b5b2b453-4c05-4a53-bedf-319e648831fd\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") "
Sep 11 15:05:27.146109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.145605    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-dshm\") pod \"b5b2b453-4c05-4a53-bedf-319e648831fd\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") "
Sep 11 15:05:27.146109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.145641    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5b2b453-4c05-4a53-bedf-319e648831fd" (UID: "b5b2b453-4c05-4a53-bedf-319e648831fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.146109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.145678    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b2b453-4c05-4a53-bedf-319e648831fd-tls-certs\") pod \"b5b2b453-4c05-4a53-bedf-319e648831fd\" (UID: \"b5b2b453-4c05-4a53-bedf-319e648831fd\") "
Sep 11 15:05:27.146109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.145810    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5b2b453-4c05-4a53-bedf-319e648831fd" (UID: "b5b2b453-4c05-4a53-bedf-319e648831fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.146109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.146049    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.146109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.146067    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.146780 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.146246    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-home" (OuterVolumeSpecName: "home") pod "b5b2b453-4c05-4a53-bedf-319e648831fd" (UID: "b5b2b453-4c05-4a53-bedf-319e648831fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.151064 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.151005    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b2b453-4c05-4a53-bedf-319e648831fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5b2b453-4c05-4a53-bedf-319e648831fd" (UID: "b5b2b453-4c05-4a53-bedf-319e648831fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:05:27.151216 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.151176    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b2b453-4c05-4a53-bedf-319e648831fd-kube-api-access-pzfc6" (OuterVolumeSpecName: "kube-api-access-pzfc6") pod "b5b2b453-4c05-4a53-bedf-319e648831fd" (UID: "b5b2b453-4c05-4a53-bedf-319e648831fd"). InnerVolumeSpecName "kube-api-access-pzfc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:05:27.151514 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.151490    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-dshm" (OuterVolumeSpecName: "dshm") pod "b5b2b453-4c05-4a53-bedf-319e648831fd" (UID: "b5b2b453-4c05-4a53-bedf-319e648831fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.167527 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.167497    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:05:27.186888 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.186858    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 15:05:27.247055 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247020    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-home\") pod \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") "
Sep 11 15:05:27.247246 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247080    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mljhd\" (UniqueName: \"kubernetes.io/projected/c46b1110-8118-457d-a50c-fad9d9aee29a-kube-api-access-mljhd\") pod \"c46b1110-8118-457d-a50c-fad9d9aee29a\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") "
Sep 11 15:05:27.247246 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247129    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-model-cache\") pod \"c46b1110-8118-457d-a50c-fad9d9aee29a\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") "
Sep 11 15:05:27.247246 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247156    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6qwt\" (UniqueName: \"kubernetes.io/projected/1f02f1ac-5778-4554-bfe0-8d03a17480dc-kube-api-access-c6qwt\") pod \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") "
Sep 11 15:05:27.247246 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247196    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tls-certs\") pod \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") "
Sep 11 15:05:27.247246 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247241    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-home\") pod \"c46b1110-8118-457d-a50c-fad9d9aee29a\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") "
Sep 11 15:05:27.247588 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247270    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46b1110-8118-457d-a50c-fad9d9aee29a-tls-certs\") pod \"c46b1110-8118-457d-a50c-fad9d9aee29a\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") "
Sep 11 15:05:27.247588 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247315    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-tmp-dir\") pod \"c46b1110-8118-457d-a50c-fad9d9aee29a\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") "
Sep 11 15:05:27.247588 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247356    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tmp-dir\") pod \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") "
Sep 11 15:05:27.247588 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247354    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-home" (OuterVolumeSpecName: "home") pod "1f02f1ac-5778-4554-bfe0-8d03a17480dc" (UID: "1f02f1ac-5778-4554-bfe0-8d03a17480dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.247588 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247388    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-dshm\") pod \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") "
Sep 11 15:05:27.247588 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247428    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c46b1110-8118-457d-a50c-fad9d9aee29a" (UID: "c46b1110-8118-457d-a50c-fad9d9aee29a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.247588 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247441    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-model-cache\") pod \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\" (UID: \"1f02f1ac-5778-4554-bfe0-8d03a17480dc\") "
Sep 11 15:05:27.247588 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247528    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-dshm\") pod \"c46b1110-8118-457d-a50c-fad9d9aee29a\" (UID: \"c46b1110-8118-457d-a50c-fad9d9aee29a\") "
Sep 11 15:05:27.248011 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247650    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f02f1ac-5778-4554-bfe0-8d03a17480dc" (UID: "1f02f1ac-5778-4554-bfe0-8d03a17480dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.248011 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247868    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f02f1ac-5778-4554-bfe0-8d03a17480dc" (UID: "1f02f1ac-5778-4554-bfe0-8d03a17480dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.248011 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.247930    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-home" (OuterVolumeSpecName: "home") pod "c46b1110-8118-457d-a50c-fad9d9aee29a" (UID: "c46b1110-8118-457d-a50c-fad9d9aee29a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.248191 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.248015    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c46b1110-8118-457d-a50c-fad9d9aee29a" (UID: "c46b1110-8118-457d-a50c-fad9d9aee29a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.248191 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.248171    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.248300 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.248193    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.248300 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.248209    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzfc6\" (UniqueName: \"kubernetes.io/projected/b5b2b453-4c05-4a53-bedf-319e648831fd-kube-api-access-pzfc6\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.248300 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.248225    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b2b453-4c05-4a53-bedf-319e648831fd-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.248300 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.248238    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.248300 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.248249    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.248597 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.248395    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b2b453-4c05-4a53-bedf-319e648831fd-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.248597 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.248411    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.248597 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.248425    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.248597 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.248436    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.249962 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.249933    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-dshm" (OuterVolumeSpecName: "dshm") pod "1f02f1ac-5778-4554-bfe0-8d03a17480dc" (UID: "1f02f1ac-5778-4554-bfe0-8d03a17480dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.250119 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.249928    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f02f1ac-5778-4554-bfe0-8d03a17480dc" (UID: "1f02f1ac-5778-4554-bfe0-8d03a17480dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:05:27.250119 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.250090    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c46b1110-8118-457d-a50c-fad9d9aee29a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c46b1110-8118-457d-a50c-fad9d9aee29a" (UID: "c46b1110-8118-457d-a50c-fad9d9aee29a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:05:27.250119 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.250103    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f02f1ac-5778-4554-bfe0-8d03a17480dc-kube-api-access-c6qwt" (OuterVolumeSpecName: "kube-api-access-c6qwt") pod "1f02f1ac-5778-4554-bfe0-8d03a17480dc" (UID: "1f02f1ac-5778-4554-bfe0-8d03a17480dc"). InnerVolumeSpecName "kube-api-access-c6qwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:05:27.250390 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.250367    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-dshm" (OuterVolumeSpecName: "dshm") pod "c46b1110-8118-457d-a50c-fad9d9aee29a" (UID: "c46b1110-8118-457d-a50c-fad9d9aee29a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.250881 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.250855    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46b1110-8118-457d-a50c-fad9d9aee29a-kube-api-access-mljhd" (OuterVolumeSpecName: "kube-api-access-mljhd") pod "c46b1110-8118-457d-a50c-fad9d9aee29a" (UID: "c46b1110-8118-457d-a50c-fad9d9aee29a"). InnerVolumeSpecName "kube-api-access-mljhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:05:27.278435 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.278395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:27.349883 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.349847    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f02f1ac-5778-4554-bfe0-8d03a17480dc-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.349883 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.349875    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46b1110-8118-457d-a50c-fad9d9aee29a-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.349883 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.349886    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mljhd\" (UniqueName: \"kubernetes.io/projected/c46b1110-8118-457d-a50c-fad9d9aee29a-kube-api-access-mljhd\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.350123 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.349896    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6qwt\" (UniqueName: \"kubernetes.io/projected/1f02f1ac-5778-4554-bfe0-8d03a17480dc-kube-api-access-c6qwt\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.350123 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.349906    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f02f1ac-5778-4554-bfe0-8d03a17480dc-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.350123 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.349914    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46b1110-8118-457d-a50c-fad9d9aee29a-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.962094 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.962059    2559 generic.go:356] "Generic (PLEG): container finished" podID="b5b2b453-4c05-4a53-bedf-319e648831fd" containerID="3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4" exitCode=0
Sep 11 15:05:27.962606 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.962101    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"
Sep 11 15:05:27.962606 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.962146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf" event={"ID":"b5b2b453-4c05-4a53-bedf-319e648831fd","Type":"ContainerDied","Data":"3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4"}
Sep 11 15:05:27.962606 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.962187    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf" event={"ID":"b5b2b453-4c05-4a53-bedf-319e648831fd","Type":"ContainerDied","Data":"32dfd74449332b6a7ae20340a3445d9e6b9125b59b48cdae2a9a475a57096b3e"}
Sep 11 15:05:27.962606 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.962207    2559 scope.go:117] "RemoveContainer" containerID="3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4"
Sep 11 15:05:27.963701 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.963675    2559 generic.go:356] "Generic (PLEG): container finished" podID="1f02f1ac-5778-4554-bfe0-8d03a17480dc" containerID="abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846" exitCode=0
Sep 11 15:05:27.963774 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.963717    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"
Sep 11 15:05:27.963774 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.963729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr" event={"ID":"1f02f1ac-5778-4554-bfe0-8d03a17480dc","Type":"ContainerDied","Data":"abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846"}
Sep 11 15:05:27.963887 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.963781    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr" event={"ID":"1f02f1ac-5778-4554-bfe0-8d03a17480dc","Type":"ContainerDied","Data":"e43c00cae392a263db8eb767726366bc6dad57e173dfba72731f86a3f2c662c9"}
Sep 11 15:05:27.965248 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.965226    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2" event={"ID":"c46b1110-8118-457d-a50c-fad9d9aee29a","Type":"ContainerDied","Data":"ed5d625ba65bc278154ccb7d81fbef45de56f83432eeab4ffab59a8e4549890d"}
Sep 11 15:05:27.965373 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.965232    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"
Sep 11 15:05:27.972013 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.971863    2559 scope.go:117] "RemoveContainer" containerID="3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4"
Sep 11 15:05:27.972237 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:05:27.972213    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4\": container with ID starting with 3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4 not found: ID does not exist" containerID="3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4"
Sep 11 15:05:27.972379 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.972249    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4"} err="failed to get container status \"3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4\": rpc error: code = NotFound desc = could not find container \"3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4\": container with ID starting with 3459563153f0848c580780b6a9169844acb4214e9fd3f997b7893b422b4548f4 not found: ID does not exist"
Sep 11 15:05:27.972379 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.972272    2559 scope.go:117] "RemoveContainer" containerID="abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846"
Sep 11 15:05:27.981261 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.981237    2559 scope.go:117] "RemoveContainer" containerID="abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846"
Sep 11 15:05:27.981598 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:05:27.981578    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846\": container with ID starting with abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846 not found: ID does not exist" containerID="abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846"
Sep 11 15:05:27.981672 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.981605    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846"} err="failed to get container status \"abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846\": rpc error: code = NotFound desc = could not find container \"abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846\": container with ID starting with abdae6aa24c224c7c89dcc2ebfcf44455c78271cd4527c0177a52a3cbbfb9846 not found: ID does not exist"
Sep 11 15:05:27.981672 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.981623    2559 scope.go:117] "RemoveContainer" containerID="82a7e867d0e1a5ef2e80a2b840d3ae9c6b8edd2c603b85d841cf0ed89a1b42ea"
Sep 11 15:05:27.987150 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.987128    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"]
Sep 11 15:05:27.992612 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:27.992585    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf"]
Sep 11 15:05:28.005830 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:28.005800    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"]
Sep 11 15:05:28.009879 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:28.009854    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/simulator-w0-e7ca56-kserve-b76fdf748-dmcbr"]
Sep 11 15:05:28.022829 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:28.022800    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"]
Sep 11 15:05:28.028416 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:28.028389    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-2-w0-e7ca56-kserve-69fb9b7db7-n44s2"]
Sep 11 15:05:28.050649 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:28.050596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:28.180836 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:28.180754    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f02f1ac-5778-4554-bfe0-8d03a17480dc" path="/var/lib/kubelet/pods/1f02f1ac-5778-4554-bfe0-8d03a17480dc/volumes"
Sep 11 15:05:28.181166 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:28.181153    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b2b453-4c05-4a53-bedf-319e648831fd" path="/var/lib/kubelet/pods/b5b2b453-4c05-4a53-bedf-319e648831fd/volumes"
Sep 11 15:05:28.181521 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:28.181508    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c46b1110-8118-457d-a50c-fad9d9aee29a" path="/var/lib/kubelet/pods/c46b1110-8118-457d-a50c-fad9d9aee29a/volumes"
Sep 11 15:05:28.934920 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:28.934885    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:29.567897 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:29.567858    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:29.725232 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:29.725182    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:29.937388 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:29.937296    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:30.281046 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:30.281008    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:30.463959 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:30.463918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:30.613318 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:30.613229    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:31.151420 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:31.151383    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:31.250850 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:31.250817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:31.441248 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:31.441151    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:31.659558 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:31.659526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:32.084360 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:32.084309    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e7ca56/premium-w0-e7ca56-kserve-65f5459c4f-rgdjf" podUID="b5b2b453-4c05-4a53-bedf-319e648831fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Sep 11 15:05:32.550220 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:32.550174    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:33.293120 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:33.293014    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:33.681795 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:33.681705    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:33.785467 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:33.785429    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:34.219444 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:34.219412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:35.030482 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:35.030422    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:35.931611 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:35.931577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:36.585664 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:36.585629    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:36.685302 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:36.685261    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:37.394136 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:37.394104    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:38.195946 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.195909    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:38.879828 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.879788    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"]
Sep 11 15:05:38.880745 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.880713    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c46b1110-8118-457d-a50c-fad9d9aee29a" containerName="main"
Sep 11 15:05:38.880745 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.880739    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c46b1110-8118-457d-a50c-fad9d9aee29a" containerName="main"
Sep 11 15:05:38.880892 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.880763    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f02f1ac-5778-4554-bfe0-8d03a17480dc" containerName="main"
Sep 11 15:05:38.880892 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.880772    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f02f1ac-5778-4554-bfe0-8d03a17480dc" containerName="main"
Sep 11 15:05:38.880892 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.880801    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b2b453-4c05-4a53-bedf-319e648831fd" containerName="main"
Sep 11 15:05:38.880892 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.880809    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b2b453-4c05-4a53-bedf-319e648831fd" containerName="main"
Sep 11 15:05:38.881018 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.880910    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f02f1ac-5778-4554-bfe0-8d03a17480dc" containerName="main"
Sep 11 15:05:38.881018 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.880924    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b2b453-4c05-4a53-bedf-319e648831fd" containerName="main"
Sep 11 15:05:38.881018 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.880936    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c46b1110-8118-457d-a50c-fad9d9aee29a" containerName="main"
Sep 11 15:05:38.886050 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.886025    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:38.888867 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.888846    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-14a808-serving-cert\""
Sep 11 15:05:38.891502 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.890525    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"]
Sep 11 15:05:38.977259 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.977219    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-openshift-service-ca\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:38.977429 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.977273    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt7zv\" (UniqueName: \"kubernetes.io/projected/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-kube-api-access-wt7zv\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:38.977548 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.977330    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-metrics-certs\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:38.977548 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:38.977541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-maas-api-tls\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:39.078794 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.078751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-openshift-service-ca\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:39.078794 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.078797    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt7zv\" (UniqueName: \"kubernetes.io/projected/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-kube-api-access-wt7zv\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:39.079005 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.078833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-metrics-certs\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:39.079005 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.078865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-maas-api-tls\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:39.079077 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:05:39.079005    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-serving-cert: secret "maas-api-e2e-api-b-14a808-serving-cert" not found
Sep 11 15:05:39.079109 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:05:39.079078    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-maas-api-tls podName:9e7b76f9-cedf-4d50-a8d7-fcb98f294079 nodeName:}" failed. No retries permitted until 2026-09-11 15:05:39.579057845 +0000 UTC m=+1545.901686192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-maas-api-tls") pod "maas-api-e2e-api-b-14a808-67b9594dcb-w26kw" (UID: "9e7b76f9-cedf-4d50-a8d7-fcb98f294079") : secret "maas-api-e2e-api-b-14a808-serving-cert" not found
Sep 11 15:05:39.079533 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.079494    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-openshift-service-ca\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:39.081507 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.081484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-metrics-certs\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:39.087516 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.087485    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt7zv\" (UniqueName: \"kubernetes.io/projected/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-kube-api-access-wt7zv\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:39.321804 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.321763    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:05:39.583916 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.583826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-maas-api-tls\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:39.586272 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.586243    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-maas-api-tls\") pod \"maas-api-e2e-api-b-14a808-67b9594dcb-w26kw\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:39.742271 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.742219    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:39.799169 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:39.799127    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:40.082297 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:40.082266    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"]
Sep 11 15:05:40.084336 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:05:40.084293    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7b76f9_cedf_4d50_a8d7_fcb98f294079.slice/crio-97bfe52dc624886c9a48a9bc7515382e94153f3607ce50073baecc58ab364fe7 WatchSource:0}: Error finding container 97bfe52dc624886c9a48a9bc7515382e94153f3607ce50073baecc58ab364fe7: Status 404 returned error can't find the container with id 97bfe52dc624886c9a48a9bc7515382e94153f3607ce50073baecc58ab364fe7
Sep 11 15:05:41.042697 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:41.042654    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw" event={"ID":"9e7b76f9-cedf-4d50-a8d7-fcb98f294079","Type":"ContainerStarted","Data":"df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1"}
Sep 11 15:05:41.043209 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:41.042752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw" event={"ID":"9e7b76f9-cedf-4d50-a8d7-fcb98f294079","Type":"ContainerStarted","Data":"97bfe52dc624886c9a48a9bc7515382e94153f3607ce50073baecc58ab364fe7"}
Sep 11 15:05:41.043209 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:41.042777    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:41.063726 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:41.063664    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw" podStartSLOduration=2.598384074 podStartE2EDuration="3.063635836s" totalImagesPullingTime="465.251762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:05:41.062217677 +0000 UTC m=+1547.384846046" watchObservedRunningTime="2026-09-11 15:05:41.063635836 +0000 UTC m=+1547.386264205"
Sep 11 15:05:47.054914 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:47.054875    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:05:48.863011 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:48.862902    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:48.964936 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:48.964899    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:49.825615 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:49.825575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:05:55.763829 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:05:55.763792    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:01.333622 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:01.333580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:02.351029 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:02.350993    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:03.151545 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:03.151505    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:03.754057 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:03.754025    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:04.244227 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:04.244188    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:05.186410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:05.186371    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:06.008299 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:06.008261    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:07.113977 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:07.113935    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:08.042209 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:08.042177    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:08.828310 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:08.828270    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:09.331985 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:09.331947    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:06:09.710835 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:09.710738    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:10.561028 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:10.560991    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:11.425030 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:11.424975    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:12.350488 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:12.350415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:13.324723 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:13.324686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:14.143402 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:14.143359    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:15.841604 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:15.841560    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:16.379265 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:16.379225    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:17.125511 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:17.125437    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:18.008411 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:18.008361    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:18.766726 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:18.766678    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:19.460128 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:19.460096    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:20.309264 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:20.309211    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:20.816881 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:20.816844    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"]
Sep 11 15:06:20.841660 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:20.841624    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"]
Sep 11 15:06:20.913934 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:20.913898    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:21.097684 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:21.097598    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:21.829076 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:21.829040    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:22.729170 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:22.729127    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"]
Sep 11 15:06:22.729392 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:22.729359    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s" podUID="2ed86dcf-37ce-4e4a-9d2e-c24a233f961b" containerName="maas-api" containerID="cri-o://76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d" gracePeriod=30
Sep 11 15:06:22.762269 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:22.762231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:22.850303 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:22.850263    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s" podUID="2ed86dcf-37ce-4e4a-9d2e-c24a233f961b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.75:8443/health\": dial tcp 10.132.0.75:8443: connect: connection refused"
Sep 11 15:06:22.859670 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:22.859635    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:22.958833 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:22.958780    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:22.982050 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:22.981979    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:06:23.060590 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.060554    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:23.151347 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.151310    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rsgl\" (UniqueName: \"kubernetes.io/projected/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-kube-api-access-2rsgl\") pod \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") "
Sep 11 15:06:23.151548 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.151379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-maas-api-tls\") pod \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") "
Sep 11 15:06:23.151548 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.151439    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-openshift-service-ca\") pod \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") "
Sep 11 15:06:23.151691 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.151580    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-metrics-certs\") pod \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\" (UID: \"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b\") "
Sep 11 15:06:23.151874 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.151850    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ed86dcf-37ce-4e4a-9d2e-c24a233f961b" (UID: "2ed86dcf-37ce-4e4a-9d2e-c24a233f961b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:06:23.151990 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.151973    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:23.153708 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.153681    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-kube-api-access-2rsgl" (OuterVolumeSpecName: "kube-api-access-2rsgl") pod "2ed86dcf-37ce-4e4a-9d2e-c24a233f961b" (UID: "2ed86dcf-37ce-4e4a-9d2e-c24a233f961b"). InnerVolumeSpecName "kube-api-access-2rsgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:23.154317 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.154285    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ed86dcf-37ce-4e4a-9d2e-c24a233f961b" (UID: "2ed86dcf-37ce-4e4a-9d2e-c24a233f961b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:06:23.154404 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.154309    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2ed86dcf-37ce-4e4a-9d2e-c24a233f961b" (UID: "2ed86dcf-37ce-4e4a-9d2e-c24a233f961b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:06:23.253297 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.253204    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:23.253297 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.253249    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rsgl\" (UniqueName: \"kubernetes.io/projected/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-kube-api-access-2rsgl\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:23.253297 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.253260    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:23.275053 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.275015    2559 generic.go:356] "Generic (PLEG): container finished" podID="2ed86dcf-37ce-4e4a-9d2e-c24a233f961b" containerID="76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d" exitCode=0
Sep 11 15:06:23.275225 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.275059    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"
Sep 11 15:06:23.275225 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.275102    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s" event={"ID":"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b","Type":"ContainerDied","Data":"76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d"}
Sep 11 15:06:23.275225 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.275157    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s" event={"ID":"2ed86dcf-37ce-4e4a-9d2e-c24a233f961b","Type":"ContainerDied","Data":"1ffcd95de85de5a38ead5207336e66e88ffe6a06384b486044a75476fcd484d8"}
Sep 11 15:06:23.275225 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.275184    2559 scope.go:117] "RemoveContainer" containerID="76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d"
Sep 11 15:06:23.284512 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.284490    2559 scope.go:117] "RemoveContainer" containerID="76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d"
Sep 11 15:06:23.284796 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:06:23.284778    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d\": container with ID starting with 76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d not found: ID does not exist" containerID="76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d"
Sep 11 15:06:23.284863 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.284808    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d"} err="failed to get container status \"76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d\": rpc error: code = NotFound desc = could not find container \"76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d\": container with ID starting with 76d30d8a6085bf87575e886948f183ce6ff81baf2940324e1d836fa37b2afa0d not found: ID does not exist"
Sep 11 15:06:23.298527 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.298490    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"]
Sep 11 15:06:23.302438 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.302412    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54b7f9-559dc5988-bt82s"]
Sep 11 15:06:23.392870 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.392835    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:23.485557 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:23.485522    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:24.180678 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:24.180644    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ed86dcf-37ce-4e4a-9d2e-c24a233f961b" path="/var/lib/kubelet/pods/2ed86dcf-37ce-4e4a-9d2e-c24a233f961b/volumes"
Sep 11 15:06:24.417150 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:24.417089    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:25.152041 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:25.151997    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:25.877580 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:25.877538    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:26.756244 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:26.756192    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:27.535356 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:27.535314    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:29.917277 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:29.917234    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:29.987996 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:29.987949    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:30.206746 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:30.206660    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:30.777335 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:30.777288    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:31.898279 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:31.898240    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:32.533830 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:32.533785    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:32.534364 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:32.534333    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k"]
Sep 11 15:06:32.536449 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:32.536425    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-837632-29818980-zzb2k"]
Sep 11 15:06:32.610711 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:32.610676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:32.768832 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:32.768794    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:32.906900 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:32.906812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:33.084303 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:33.084264    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:33.184258 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:33.184164    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:33.456997 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:33.456914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:34.028285 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:34.028233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:34.180816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:34.180781    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b55a9325-2818-4247-b865-d665347f86e4" path="/var/lib/kubelet/pods/b55a9325-2818-4247-b865-d665347f86e4/volumes"
Sep 11 15:06:34.810635 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:34.810595    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:35.523385 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:35.523344    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:35.836222 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:35.836130    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2" podUID="90ab43ad-10fc-4be9-9f30-e1d486dc93ea" containerName="main" containerID="cri-o://1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e" gracePeriod=45
Sep 11 15:06:35.860724 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:35.860680    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk" podUID="4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" containerName="main" containerID="cri-o://5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1" gracePeriod=45
Sep 11 15:06:36.135644 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.135619    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:06:36.138213 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.138193    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 15:06:36.176192 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176159    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-home\") pod \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") "
Sep 11 15:06:36.176192 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176207    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-dshm\") pod \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") "
Sep 11 15:06:36.176445 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176236    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-home\") pod \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") "
Sep 11 15:06:36.176445 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176281    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-dshm\") pod \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") "
Sep 11 15:06:36.176445 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176327    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42shw\" (UniqueName: \"kubernetes.io/projected/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-kube-api-access-42shw\") pod \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") "
Sep 11 15:06:36.176445 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176377    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tmp-dir\") pod \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") "
Sep 11 15:06:36.176445 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176426    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-model-cache\") pod \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") "
Sep 11 15:06:36.176743 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176482    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-model-cache\") pod \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") "
Sep 11 15:06:36.176743 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176491    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-home" (OuterVolumeSpecName: "home") pod "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" (UID: "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:36.176743 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176558    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tmp-dir\") pod \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") "
Sep 11 15:06:36.176743 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176585    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmsl4\" (UniqueName: \"kubernetes.io/projected/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-kube-api-access-vmsl4\") pod \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") "
Sep 11 15:06:36.176743 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176617    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tls-certs\") pod \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\" (UID: \"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c\") "
Sep 11 15:06:36.176743 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176650    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tls-certs\") pod \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\" (UID: \"90ab43ad-10fc-4be9-9f30-e1d486dc93ea\") "
Sep 11 15:06:36.176743 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176707    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-home" (OuterVolumeSpecName: "home") pod "90ab43ad-10fc-4be9-9f30-e1d486dc93ea" (UID: "90ab43ad-10fc-4be9-9f30-e1d486dc93ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:36.176743 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176715    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" (UID: "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:36.177144 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.176918    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90ab43ad-10fc-4be9-9f30-e1d486dc93ea" (UID: "90ab43ad-10fc-4be9-9f30-e1d486dc93ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:36.177144 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.177081    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.177144 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.177099    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.177144 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.177113    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.177144 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.177127    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.177144 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.177081    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" (UID: "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:36.179023 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.178990    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "90ab43ad-10fc-4be9-9f30-e1d486dc93ea" (UID: "90ab43ad-10fc-4be9-9f30-e1d486dc93ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:36.179983 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.179876    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90ab43ad-10fc-4be9-9f30-e1d486dc93ea" (UID: "90ab43ad-10fc-4be9-9f30-e1d486dc93ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:06:36.179983 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.179952    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-dshm" (OuterVolumeSpecName: "dshm") pod "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" (UID: "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:36.181149 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.181071    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-dshm" (OuterVolumeSpecName: "dshm") pod "90ab43ad-10fc-4be9-9f30-e1d486dc93ea" (UID: "90ab43ad-10fc-4be9-9f30-e1d486dc93ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:36.181259 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.181196    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-kube-api-access-vmsl4" (OuterVolumeSpecName: "kube-api-access-vmsl4") pod "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" (UID: "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c"). InnerVolumeSpecName "kube-api-access-vmsl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:36.181325 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.181292    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-kube-api-access-42shw" (OuterVolumeSpecName: "kube-api-access-42shw") pod "90ab43ad-10fc-4be9-9f30-e1d486dc93ea" (UID: "90ab43ad-10fc-4be9-9f30-e1d486dc93ea"). InnerVolumeSpecName "kube-api-access-42shw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:36.181486 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.181442    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" (UID: "4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:06:36.278168 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.278130    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.278168 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.278162    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42shw\" (UniqueName: \"kubernetes.io/projected/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-kube-api-access-42shw\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.278168 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.278174    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.278408 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.278183    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.278408 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.278191    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmsl4\" (UniqueName: \"kubernetes.io/projected/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-kube-api-access-vmsl4\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.278408 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.278199    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.278408 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.278207    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.278408 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.278214    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ab43ad-10fc-4be9-9f30-e1d486dc93ea-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:36.313964 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.313929    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:36.341639 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.341606    2559 generic.go:356] "Generic (PLEG): container finished" podID="4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" containerID="5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1" exitCode=0
Sep 11 15:06:36.341810 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.341654    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"
Sep 11 15:06:36.341810 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.341698    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk" event={"ID":"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c","Type":"ContainerDied","Data":"5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1"}
Sep 11 15:06:36.341810 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.341744    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk" event={"ID":"4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c","Type":"ContainerDied","Data":"e98c3541999297d04784ab74f7aec99abdda48e10d9376111b0fa6329e6008d8"}
Sep 11 15:06:36.341810 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.341766    2559 scope.go:117] "RemoveContainer" containerID="5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1"
Sep 11 15:06:36.343135 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.343115    2559 generic.go:356] "Generic (PLEG): container finished" podID="90ab43ad-10fc-4be9-9f30-e1d486dc93ea" containerID="1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e" exitCode=0
Sep 11 15:06:36.343241 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.343166    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"
Sep 11 15:06:36.343241 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.343187    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2" event={"ID":"90ab43ad-10fc-4be9-9f30-e1d486dc93ea","Type":"ContainerDied","Data":"1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e"}
Sep 11 15:06:36.343241 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.343220    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2" event={"ID":"90ab43ad-10fc-4be9-9f30-e1d486dc93ea","Type":"ContainerDied","Data":"8fd9c6e7b9555363735296c8d32cf9111870b7b1876249aa673d33ee2fcb2873"}
Sep 11 15:06:36.350576 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.350554    2559 scope.go:117] "RemoveContainer" containerID="5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1"
Sep 11 15:06:36.350876 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:06:36.350855    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1\": container with ID starting with 5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1 not found: ID does not exist" containerID="5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1"
Sep 11 15:06:36.350936 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.350884    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1"} err="failed to get container status \"5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1\": rpc error: code = NotFound desc = could not find container \"5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1\": container with ID starting with 5efca8b7abcec8e3be2cd20b35c6bd14bbfbdd3c293971b695b71124286a6aa1 not found: ID does not exist"
Sep 11 15:06:36.350936 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.350917    2559 scope.go:117] "RemoveContainer" containerID="1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e"
Sep 11 15:06:36.358747 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.358676    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"]
Sep 11 15:06:36.360797 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.360775    2559 scope.go:117] "RemoveContainer" containerID="1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e"
Sep 11 15:06:36.361170 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:06:36.361139    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e\": container with ID starting with 1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e not found: ID does not exist" containerID="1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e"
Sep 11 15:06:36.361287 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.361179    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e"} err="failed to get container status \"1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e\": rpc error: code = NotFound desc = could not find container \"1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e\": container with ID starting with 1d4d5857c3335a790c4dc81684946872e716e7e45d5c7d7859d7978d23e5ec9e not found: ID does not exist"
Sep 11 15:06:36.363893 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.363871    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-837632/simulator-w4-837632-kserve-74594cf674-67mmk"]
Sep 11 15:06:36.379239 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.379168    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"]
Sep 11 15:06:36.382300 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:36.382273    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-2-w4-837632-kserve-68764589fc-r9pk2"]
Sep 11 15:06:37.131407 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:37.131362    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:37.433001 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:37.432911    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:06:37.861519 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:37.861479    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:38.181844 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:38.181750    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" path="/var/lib/kubelet/pods/4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c/volumes"
Sep 11 15:06:38.182319 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:38.182295    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90ab43ad-10fc-4be9-9f30-e1d486dc93ea" path="/var/lib/kubelet/pods/90ab43ad-10fc-4be9-9f30-e1d486dc93ea/volumes"
Sep 11 15:06:39.882836 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:39.882797    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:39.984568 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:39.984524    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:40.224600 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:40.224565    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:40.738648 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:40.738614    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:42.112306 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:42.112244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:46.420792 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.420704    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"]
Sep 11 15:06:46.423267 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.420960    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw" podUID="9e7b76f9-cedf-4d50-a8d7-fcb98f294079" containerName="maas-api" containerID="cri-o://df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1" gracePeriod=30
Sep 11 15:06:46.454815 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.454780    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:46.529598 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.529560    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:46.602278 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.602141    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:46.676789 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.676718    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:06:46.701083 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.701050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:46.787034 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.787005    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt7zv\" (UniqueName: \"kubernetes.io/projected/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-kube-api-access-wt7zv\") pod \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") "
Sep 11 15:06:46.787220 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.787072    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-maas-api-tls\") pod \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") "
Sep 11 15:06:46.787220 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.787134    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-metrics-certs\") pod \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") "
Sep 11 15:06:46.787220 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.787206    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-openshift-service-ca\") pod \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\" (UID: \"9e7b76f9-cedf-4d50-a8d7-fcb98f294079\") "
Sep 11 15:06:46.787756 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.787709    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e7b76f9-cedf-4d50-a8d7-fcb98f294079" (UID: "9e7b76f9-cedf-4d50-a8d7-fcb98f294079"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:06:46.789671 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.789608    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-kube-api-access-wt7zv" (OuterVolumeSpecName: "kube-api-access-wt7zv") pod "9e7b76f9-cedf-4d50-a8d7-fcb98f294079" (UID: "9e7b76f9-cedf-4d50-a8d7-fcb98f294079"). InnerVolumeSpecName "kube-api-access-wt7zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:46.789831 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.789719    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e7b76f9-cedf-4d50-a8d7-fcb98f294079" (UID: "9e7b76f9-cedf-4d50-a8d7-fcb98f294079"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:06:46.789831 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.789755    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9e7b76f9-cedf-4d50-a8d7-fcb98f294079" (UID: "9e7b76f9-cedf-4d50-a8d7-fcb98f294079"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:06:46.878177 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.878131    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:46.888411 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.888379    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:46.888411 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.888407    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:46.888411 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.888416    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:46.888661 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:46.888427    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt7zv\" (UniqueName: \"kubernetes.io/projected/9e7b76f9-cedf-4d50-a8d7-fcb98f294079-kube-api-access-wt7zv\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:06:47.104258 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.104221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:47.195169 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.195133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:47.402943 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.402852    2559 generic.go:356] "Generic (PLEG): container finished" podID="9e7b76f9-cedf-4d50-a8d7-fcb98f294079" containerID="df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1" exitCode=0
Sep 11 15:06:47.402943 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.402908    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"
Sep 11 15:06:47.403153 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.402937    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw" event={"ID":"9e7b76f9-cedf-4d50-a8d7-fcb98f294079","Type":"ContainerDied","Data":"df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1"}
Sep 11 15:06:47.403153 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.402993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw" event={"ID":"9e7b76f9-cedf-4d50-a8d7-fcb98f294079","Type":"ContainerDied","Data":"97bfe52dc624886c9a48a9bc7515382e94153f3607ce50073baecc58ab364fe7"}
Sep 11 15:06:47.403153 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.403015    2559 scope.go:117] "RemoveContainer" containerID="df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1"
Sep 11 15:06:47.412820 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.412800    2559 scope.go:117] "RemoveContainer" containerID="df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1"
Sep 11 15:06:47.413132 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:06:47.413112    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1\": container with ID starting with df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1 not found: ID does not exist" containerID="df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1"
Sep 11 15:06:47.413208 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.413142    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1"} err="failed to get container status \"df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1\": rpc error: code = NotFound desc = could not find container \"df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1\": container with ID starting with df39d8cf560f6e5ea71011615c9ddab3b5d62102a50fcd754262430bb79bfdb1 not found: ID does not exist"
Sep 11 15:06:47.425277 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.425247    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"]
Sep 11 15:06:47.430220 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.430198    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14a808-67b9594dcb-w26kw"]
Sep 11 15:06:47.966672 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:47.966634    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:48.180516 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:48.180471    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e7b76f9-cedf-4d50-a8d7-fcb98f294079" path="/var/lib/kubelet/pods/9e7b76f9-cedf-4d50-a8d7-fcb98f294079/volumes"
Sep 11 15:06:48.691695 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:48.691647    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:49.628516 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:49.628475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:50.352754 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:50.352713    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:53.877923 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:53.877879    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:53.962733 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:53.962673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:54.185011 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:54.184906    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:54.410558 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:54.410519    2559 scope.go:117] "RemoveContainer" containerID="837be2e91986db4bdd5c01acff38839c91e1fb8b8a1ddc9bdf42506e1f59de76"
Sep 11 15:06:54.419083 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:54.419063    2559 scope.go:117] "RemoveContainer" containerID="2d2de6e8a5d39dc72c4f1a519a839a39da06560df574befcb0d213c8fc4f3b2f"
Sep 11 15:06:55.513512 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:55.513480    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:56.209512 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:56.209478    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:56.288510 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:56.288480    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:56.378477 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:56.378420    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:57.181126 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:57.181085    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:57.900665 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:57.900623    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:06:58.608159 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:06:58.608121    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:02.544576 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:02.544534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:07:04.628151 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:04.628108    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:04.728959 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:04.728921    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:05.112004 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.111964    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"]
Sep 11 15:07:05.112269 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.112244    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k" podUID="7dfc0283-a6ef-4658-9584-ad55c27104ac" containerName="maas-api" containerID="cri-o://a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad" gracePeriod=30
Sep 11 15:07:05.151139 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.151099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:05.249229 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.249194    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:05.349182 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.349127    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:05.362629 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.362552    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 15:07:05.366314 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.366288    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-maas-api-tls\") pod \"7dfc0283-a6ef-4658-9584-ad55c27104ac\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") "
Sep 11 15:07:05.366472 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.366328    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f4dr\" (UniqueName: \"kubernetes.io/projected/7dfc0283-a6ef-4658-9584-ad55c27104ac-kube-api-access-2f4dr\") pod \"7dfc0283-a6ef-4658-9584-ad55c27104ac\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") "
Sep 11 15:07:05.366472 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.366416    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-metrics-certs\") pod \"7dfc0283-a6ef-4658-9584-ad55c27104ac\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") "
Sep 11 15:07:05.368694 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.368655    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7dfc0283-a6ef-4658-9584-ad55c27104ac" (UID: "7dfc0283-a6ef-4658-9584-ad55c27104ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:07:05.368694 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.368679    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7dfc0283-a6ef-4658-9584-ad55c27104ac" (UID: "7dfc0283-a6ef-4658-9584-ad55c27104ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:07:05.368929 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.368752    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfc0283-a6ef-4658-9584-ad55c27104ac-kube-api-access-2f4dr" (OuterVolumeSpecName: "kube-api-access-2f4dr") pod "7dfc0283-a6ef-4658-9584-ad55c27104ac" (UID: "7dfc0283-a6ef-4658-9584-ad55c27104ac"). InnerVolumeSpecName "kube-api-access-2f4dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:07:05.449657 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.449617    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:05.467854 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.467817    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfc0283-a6ef-4658-9584-ad55c27104ac-openshift-service-ca\") pod \"7dfc0283-a6ef-4658-9584-ad55c27104ac\" (UID: \"7dfc0283-a6ef-4658-9584-ad55c27104ac\") "
Sep 11 15:07:05.468132 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.468113    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:05.468222 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.468135    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dfc0283-a6ef-4658-9584-ad55c27104ac-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:05.468222 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.468149    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f4dr\" (UniqueName: \"kubernetes.io/projected/7dfc0283-a6ef-4658-9584-ad55c27104ac-kube-api-access-2f4dr\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:05.468222 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.468171    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfc0283-a6ef-4658-9584-ad55c27104ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7dfc0283-a6ef-4658-9584-ad55c27104ac" (UID: "7dfc0283-a6ef-4658-9584-ad55c27104ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:07:05.500575 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.500517    2559 generic.go:356] "Generic (PLEG): container finished" podID="7dfc0283-a6ef-4658-9584-ad55c27104ac" containerID="a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad" exitCode=0
Sep 11 15:07:05.500741 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.500575    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"
Sep 11 15:07:05.500741 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.500599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k" event={"ID":"7dfc0283-a6ef-4658-9584-ad55c27104ac","Type":"ContainerDied","Data":"a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad"}
Sep 11 15:07:05.500741 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.500659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k" event={"ID":"7dfc0283-a6ef-4658-9584-ad55c27104ac","Type":"ContainerDied","Data":"aa64092859c4f61d35debd00d02456c025dece4f0b6c6acf87d1344a1a681efc"}
Sep 11 15:07:05.500741 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.500688    2559 scope.go:117] "RemoveContainer" containerID="a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad"
Sep 11 15:07:05.511130 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.511105    2559 scope.go:117] "RemoveContainer" containerID="a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad"
Sep 11 15:07:05.511531 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:07:05.511501    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad\": container with ID starting with a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad not found: ID does not exist" containerID="a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad"
Sep 11 15:07:05.511628 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.511543    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad"} err="failed to get container status \"a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad\": rpc error: code = NotFound desc = could not find container \"a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad\": container with ID starting with a3df188ab1a5cee74ccd477b75616296ed35130cff2bd41d5f86899679a4cbad not found: ID does not exist"
Sep 11 15:07:05.524896 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.524858    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"]
Sep 11 15:07:05.530721 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.530693    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4e39b-66bfc8968-f5g5k"]
Sep 11 15:07:05.569290 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.569253    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfc0283-a6ef-4658-9584-ad55c27104ac-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:05.792969 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.792926    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:05.891726 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.891684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:05.996863 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:05.996826    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:06.181424 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:06.181339    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dfc0283-a6ef-4658-9584-ad55c27104ac" path="/var/lib/kubelet/pods/7dfc0283-a6ef-4658-9584-ad55c27104ac/volumes"
Sep 11 15:07:06.610720 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:06.610678    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:07.438272 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:07.438238    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:08.130960 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:08.130926    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:12.061669 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:12.061618    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:12.164343 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:12.164303    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:12.344190 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:12.344100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:13.710448 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:13.710407    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:14.483132 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:14.483098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:15.312849 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:15.312803    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:16.087928 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:16.087873    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:18.219939 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:18.219897    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:07:21.283930 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.283894    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7674-64ea67"]
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284277    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed86dcf-37ce-4e4a-9d2e-c24a233f961b" containerName="maas-api"
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284290    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed86dcf-37ce-4e4a-9d2e-c24a233f961b" containerName="maas-api"
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284301    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90ab43ad-10fc-4be9-9f30-e1d486dc93ea" containerName="main"
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284307    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="90ab43ad-10fc-4be9-9f30-e1d486dc93ea" containerName="main"
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284318    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dfc0283-a6ef-4658-9584-ad55c27104ac" containerName="maas-api"
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284324    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dfc0283-a6ef-4658-9584-ad55c27104ac" containerName="maas-api"
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284331    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" containerName="main"
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284336    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" containerName="main"
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284348    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7b76f9-cedf-4d50-a8d7-fcb98f294079" containerName="maas-api"
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284354    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7b76f9-cedf-4d50-a8d7-fcb98f294079" containerName="maas-api"
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284408    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f2bdc5c-96ce-4b48-ae4a-66b96dc4259c" containerName="main"
Sep 11 15:07:21.284410 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284416    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="90ab43ad-10fc-4be9-9f30-e1d486dc93ea" containerName="main"
Sep 11 15:07:21.284848 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284425    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed86dcf-37ce-4e4a-9d2e-c24a233f961b" containerName="maas-api"
Sep 11 15:07:21.284848 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284431    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dfc0283-a6ef-4658-9584-ad55c27104ac" containerName="maas-api"
Sep 11 15:07:21.284848 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.284435    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e7b76f9-cedf-4d50-a8d7-fcb98f294079" containerName="maas-api"
Sep 11 15:07:21.288998 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.288979    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-64ea67"
Sep 11 15:07:21.292177 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.292142    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5lj77\""
Sep 11 15:07:21.294711 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.294686    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-64ea67"]
Sep 11 15:07:21.445923 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.439879    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngt4b\" (UniqueName: \"kubernetes.io/projected/7515f258-5922-4549-8c51-f7431b65ad29-kube-api-access-ngt4b\") pod \"test-curl-7674-64ea67\" (UID: \"7515f258-5922-4549-8c51-f7431b65ad29\") " pod="opendatahub/test-curl-7674-64ea67"
Sep 11 15:07:21.541189 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.541091    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngt4b\" (UniqueName: \"kubernetes.io/projected/7515f258-5922-4549-8c51-f7431b65ad29-kube-api-access-ngt4b\") pod \"test-curl-7674-64ea67\" (UID: \"7515f258-5922-4549-8c51-f7431b65ad29\") " pod="opendatahub/test-curl-7674-64ea67"
Sep 11 15:07:21.550796 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.550760    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngt4b\" (UniqueName: \"kubernetes.io/projected/7515f258-5922-4549-8c51-f7431b65ad29-kube-api-access-ngt4b\") pod \"test-curl-7674-64ea67\" (UID: \"7515f258-5922-4549-8c51-f7431b65ad29\") " pod="opendatahub/test-curl-7674-64ea67"
Sep 11 15:07:21.600245 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.600198    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-64ea67"
Sep 11 15:07:21.758739 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.758696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:21.873601 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:21.873570    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-64ea67"]
Sep 11 15:07:22.590640 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:22.590600    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-64ea67" event={"ID":"7515f258-5922-4549-8c51-f7431b65ad29","Type":"ContainerStarted","Data":"5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d"}
Sep 11 15:07:22.591119 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:22.590652    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-64ea67" event={"ID":"7515f258-5922-4549-8c51-f7431b65ad29","Type":"ContainerStarted","Data":"d3027005e852996649fd0456aa80085ee27e19f5e0a9a404c7a7ec6405d31610"}
Sep 11 15:07:22.609500 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:22.609430    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7674-64ea67" podStartSLOduration=1.6094138949999999 podStartE2EDuration="1.609413895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:07:22.607876587 +0000 UTC m=+1648.930504962" watchObservedRunningTime="2026-09-11 15:07:22.609413895 +0000 UTC m=+1648.932042264"
Sep 11 15:07:23.268642 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:23.268605    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7674-64ea67"]
Sep 11 15:07:23.273188 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:23.273158    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7674-64ea67"]
Sep 11 15:07:23.350689 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:23.350586    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:23.449854 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:23.449824    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:23.557823 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:23.557785    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:23.596517 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:07:23.596488    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7674-64ea67" container="test-curl-7674-64ea67"
Sep 11 15:07:23.650313 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:23.650228    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:23.977028 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:23.976995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:24.080623 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:24.080587    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:24.601032 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:24.600992    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7674-64ea67" podUID="7515f258-5922-4549-8c51-f7431b65ad29" containerName="test-curl-7674-64ea67" containerID="cri-o://5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d" gracePeriod=2
Sep 11 15:07:24.854892 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:24.854795    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:25.125809 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:25.125710    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:25.713897 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:25.713850    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:25.977784 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:25.977739    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"]
Sep 11 15:07:26.006878 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:26.006818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:26.791568 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:26.791533    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-64ea67"
Sep 11 15:07:26.905271 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:26.905157    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngt4b\" (UniqueName: \"kubernetes.io/projected/7515f258-5922-4549-8c51-f7431b65ad29-kube-api-access-ngt4b\") pod \"7515f258-5922-4549-8c51-f7431b65ad29\" (UID: \"7515f258-5922-4549-8c51-f7431b65ad29\") "
Sep 11 15:07:26.907298 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:26.907263    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7515f258-5922-4549-8c51-f7431b65ad29-kube-api-access-ngt4b" (OuterVolumeSpecName: "kube-api-access-ngt4b") pod "7515f258-5922-4549-8c51-f7431b65ad29" (UID: "7515f258-5922-4549-8c51-f7431b65ad29"). InnerVolumeSpecName "kube-api-access-ngt4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:07:27.006109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:27.006059    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngt4b\" (UniqueName: \"kubernetes.io/projected/7515f258-5922-4549-8c51-f7431b65ad29-kube-api-access-ngt4b\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:27.619012 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:27.618977    2559 generic.go:356] "Generic (PLEG): container finished" podID="7515f258-5922-4549-8c51-f7431b65ad29" containerID="5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d" exitCode=137
Sep 11 15:07:27.619205 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:27.619016    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-64ea67"
Sep 11 15:07:27.619205 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:27.619058    2559 scope.go:117] "RemoveContainer" containerID="5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d"
Sep 11 15:07:27.628823 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:27.628795    2559 scope.go:117] "RemoveContainer" containerID="5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d"
Sep 11 15:07:27.629198 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:07:27.629176    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d\": container with ID starting with 5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d not found: ID does not exist" containerID="5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d"
Sep 11 15:07:27.629265 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:27.629208    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d"} err="failed to get container status \"5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d\": rpc error: code = NotFound desc = could not find container \"5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d\": container with ID starting with 5e222d4f4876c5d598e9c4621b78cfd65175603c5a35ffa78dd6f74d5ae6274d not found: ID does not exist"
Sep 11 15:07:27.630352 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:07:27.630324    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-64ea67\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="7515f258-5922-4549-8c51-f7431b65ad29" pod="opendatahub/test-curl-7674-64ea67"
Sep 11 15:07:28.180707 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:28.180670    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7515f258-5922-4549-8c51-f7431b65ad29" path="/var/lib/kubelet/pods/7515f258-5922-4549-8c51-f7431b65ad29/volumes"
Sep 11 15:07:29.014667 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:29.014623    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:07:30.650977 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:30.650939    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:30.751398 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:30.751347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:30.905139 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:30.905035    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"]
Sep 11 15:07:30.905334 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:30.905262    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy" containerID="cri-o://7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7" gracePeriod=30
Sep 11 15:07:30.932236 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:30.932193    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:34.981108 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:34.981065    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:07:36.173840 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.173761    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 15:07:36.192024 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.191991    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-token\") pod \"81c2c16e-25b6-4584-9190-1fd68da51f25\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") "
Sep 11 15:07:36.192182 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192035    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-socket\") pod \"81c2c16e-25b6-4584-9190-1fd68da51f25\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") "
Sep 11 15:07:36.192182 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192068    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81c2c16e-25b6-4584-9190-1fd68da51f25-istiod-ca-cert\") pod \"81c2c16e-25b6-4584-9190-1fd68da51f25\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") "
Sep 11 15:07:36.192182 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192109    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-podinfo\") pod \"81c2c16e-25b6-4584-9190-1fd68da51f25\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") "
Sep 11 15:07:36.192182 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192155    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-certs\") pod \"81c2c16e-25b6-4584-9190-1fd68da51f25\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") "
Sep 11 15:07:36.192404 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192290    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-data\") pod \"81c2c16e-25b6-4584-9190-1fd68da51f25\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") "
Sep 11 15:07:36.192404 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192344    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwn4z\" (UniqueName: \"kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-kube-api-access-zwn4z\") pod \"81c2c16e-25b6-4584-9190-1fd68da51f25\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") "
Sep 11 15:07:36.192681 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192418    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-credential-socket\") pod \"81c2c16e-25b6-4584-9190-1fd68da51f25\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") "
Sep 11 15:07:36.192681 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192446    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-envoy\") pod \"81c2c16e-25b6-4584-9190-1fd68da51f25\" (UID: \"81c2c16e-25b6-4584-9190-1fd68da51f25\") "
Sep 11 15:07:36.192681 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192412    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81c2c16e-25b6-4584-9190-1fd68da51f25" (UID: "81c2c16e-25b6-4584-9190-1fd68da51f25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:36.192681 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192671    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81c2c16e-25b6-4584-9190-1fd68da51f25" (UID: "81c2c16e-25b6-4584-9190-1fd68da51f25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:36.192975 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192541    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81c2c16e-25b6-4584-9190-1fd68da51f25" (UID: "81c2c16e-25b6-4584-9190-1fd68da51f25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:36.192975 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192717    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c2c16e-25b6-4584-9190-1fd68da51f25-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81c2c16e-25b6-4584-9190-1fd68da51f25" (UID: "81c2c16e-25b6-4584-9190-1fd68da51f25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:07:36.192975 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192888    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-data" (OuterVolumeSpecName: "istio-data") pod "81c2c16e-25b6-4584-9190-1fd68da51f25" (UID: "81c2c16e-25b6-4584-9190-1fd68da51f25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:36.193152 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.192990    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81c2c16e-25b6-4584-9190-1fd68da51f25-istiod-ca-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:36.193152 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.193011    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:36.193152 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.193026    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-data\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:36.193152 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.193039    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-credential-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:36.193152 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.193054    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-workload-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:36.194879 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.194829    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-token" (OuterVolumeSpecName: "istio-token") pod "81c2c16e-25b6-4584-9190-1fd68da51f25" (UID: "81c2c16e-25b6-4584-9190-1fd68da51f25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:07:36.195339 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.195300    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81c2c16e-25b6-4584-9190-1fd68da51f25" (UID: "81c2c16e-25b6-4584-9190-1fd68da51f25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:36.195339 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.195318    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81c2c16e-25b6-4584-9190-1fd68da51f25" (UID: "81c2c16e-25b6-4584-9190-1fd68da51f25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 15:07:36.195743 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.195716    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-kube-api-access-zwn4z" (OuterVolumeSpecName: "kube-api-access-zwn4z") pod "81c2c16e-25b6-4584-9190-1fd68da51f25" (UID: "81c2c16e-25b6-4584-9190-1fd68da51f25"). InnerVolumeSpecName "kube-api-access-zwn4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:07:36.294699 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.294651    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-token\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:36.294699 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.294686    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-podinfo\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:36.294699 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.294696    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwn4z\" (UniqueName: \"kubernetes.io/projected/81c2c16e-25b6-4584-9190-1fd68da51f25-kube-api-access-zwn4z\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:36.294699 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.294708    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81c2c16e-25b6-4584-9190-1fd68da51f25-istio-envoy\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:36.668505 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.668425    2559 generic.go:356] "Generic (PLEG): container finished" podID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerID="7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7" exitCode=0
Sep 11 15:07:36.668693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.668509    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"
Sep 11 15:07:36.668693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.668510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" event={"ID":"81c2c16e-25b6-4584-9190-1fd68da51f25","Type":"ContainerDied","Data":"7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7"}
Sep 11 15:07:36.668693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.668579    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55" event={"ID":"81c2c16e-25b6-4584-9190-1fd68da51f25","Type":"ContainerDied","Data":"c005e5a0c626d630c248f036ba130eba382ca61ad6b97f0ba4c5f535d8fcb35e"}
Sep 11 15:07:36.668693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.668612    2559 scope.go:117] "RemoveContainer" containerID="7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7"
Sep 11 15:07:36.678602 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.678580    2559 scope.go:117] "RemoveContainer" containerID="b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf"
Sep 11 15:07:36.687723 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.687698    2559 scope.go:117] "RemoveContainer" containerID="7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7"
Sep 11 15:07:36.688115 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:07:36.688090    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7\": container with ID starting with 7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7 not found: ID does not exist" containerID="7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7"
Sep 11 15:07:36.688208 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.688124    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7"} err="failed to get container status \"7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7\": rpc error: code = NotFound desc = could not find container \"7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7\": container with ID starting with 7ef200fcbf03b39c1f4f70fcd22506b44ae62661433bba9523c851892c462fa7 not found: ID does not exist"
Sep 11 15:07:36.688208 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.688143    2559 scope.go:117] "RemoveContainer" containerID="b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf"
Sep 11 15:07:36.688415 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:07:36.688397    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf\": container with ID starting with b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf not found: ID does not exist" containerID="b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf"
Sep 11 15:07:36.688488 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.688421    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf"} err="failed to get container status \"b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf\": rpc error: code = NotFound desc = could not find container \"b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf\": container with ID starting with b897f5c59a1bfbc26924279d843ba2ccb3d89f597a6dc00afcf5d3909f8527cf not found: ID does not exist"
Sep 11 15:07:36.693468 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.693422    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"]
Sep 11 15:07:36.699602 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:36.699572    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c0e28e-openshift-default-56897975d6-bfw55"]
Sep 11 15:07:38.181213 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:38.181173    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" path="/var/lib/kubelet/pods/81c2c16e-25b6-4584-9190-1fd68da51f25/volumes"
Sep 11 15:07:39.337081 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:39.337038    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:07:40.997389 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:40.997342    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs" podUID="ab9ca370-a41f-418d-9b7e-3800b6349cc8" containerName="main" containerID="cri-o://b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a" gracePeriod=45
Sep 11 15:07:41.252819 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.252744    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:07:41.347797 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.347756    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-home\") pod \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") "
Sep 11 15:07:41.348000 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.347815    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-dshm\") pod \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") "
Sep 11 15:07:41.348000 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.347861    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwdx9\" (UniqueName: \"kubernetes.io/projected/ab9ca370-a41f-418d-9b7e-3800b6349cc8-kube-api-access-hwdx9\") pod \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") "
Sep 11 15:07:41.348000 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.347921    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tls-certs\") pod \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") "
Sep 11 15:07:41.348175 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.348106    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-home" (OuterVolumeSpecName: "home") pod "ab9ca370-a41f-418d-9b7e-3800b6349cc8" (UID: "ab9ca370-a41f-418d-9b7e-3800b6349cc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:41.348175 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.348120    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tmp-dir\") pod \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") "
Sep 11 15:07:41.348290 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.348197    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-model-cache\") pod \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\" (UID: \"ab9ca370-a41f-418d-9b7e-3800b6349cc8\") "
Sep 11 15:07:41.348354 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.348316    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab9ca370-a41f-418d-9b7e-3800b6349cc8" (UID: "ab9ca370-a41f-418d-9b7e-3800b6349cc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:41.348491 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.348446    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab9ca370-a41f-418d-9b7e-3800b6349cc8" (UID: "ab9ca370-a41f-418d-9b7e-3800b6349cc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:41.348609 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.348583    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:41.348609 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.348609    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:41.348811 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.348621    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:41.350288 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.350258    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab9ca370-a41f-418d-9b7e-3800b6349cc8" (UID: "ab9ca370-a41f-418d-9b7e-3800b6349cc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:07:41.350288 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.350276    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9ca370-a41f-418d-9b7e-3800b6349cc8-kube-api-access-hwdx9" (OuterVolumeSpecName: "kube-api-access-hwdx9") pod "ab9ca370-a41f-418d-9b7e-3800b6349cc8" (UID: "ab9ca370-a41f-418d-9b7e-3800b6349cc8"). InnerVolumeSpecName "kube-api-access-hwdx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:07:41.350483 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.350343    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-dshm" (OuterVolumeSpecName: "dshm") pod "ab9ca370-a41f-418d-9b7e-3800b6349cc8" (UID: "ab9ca370-a41f-418d-9b7e-3800b6349cc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:41.450134 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.450084    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ca370-a41f-418d-9b7e-3800b6349cc8-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:41.450134 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.450127    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab9ca370-a41f-418d-9b7e-3800b6349cc8-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:41.450134 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.450140    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwdx9\" (UniqueName: \"kubernetes.io/projected/ab9ca370-a41f-418d-9b7e-3800b6349cc8-kube-api-access-hwdx9\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:07:41.695960 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.695864    2559 generic.go:356] "Generic (PLEG): container finished" podID="ab9ca370-a41f-418d-9b7e-3800b6349cc8" containerID="b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a" exitCode=0
Sep 11 15:07:41.695960 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.695909    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"
Sep 11 15:07:41.696160 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.695952    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs" event={"ID":"ab9ca370-a41f-418d-9b7e-3800b6349cc8","Type":"ContainerDied","Data":"b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a"}
Sep 11 15:07:41.696160 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.696001    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs" event={"ID":"ab9ca370-a41f-418d-9b7e-3800b6349cc8","Type":"ContainerDied","Data":"02f95bb4070b42b7d974a62ba12a76215fbc0b6b8060a08e19152de42557e4d5"}
Sep 11 15:07:41.696160 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.696020    2559 scope.go:117] "RemoveContainer" containerID="b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a"
Sep 11 15:07:41.707888 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.707765    2559 scope.go:117] "RemoveContainer" containerID="b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a"
Sep 11 15:07:41.708401 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:07:41.708250    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a\": container with ID starting with b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a not found: ID does not exist" containerID="b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a"
Sep 11 15:07:41.708401 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.708291    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a"} err="failed to get container status \"b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a\": rpc error: code = NotFound desc = could not find container \"b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a\": container with ID starting with b98468b9a2a61c2fd1e497f2a71a46d0fc0608c601b15b1ed396804f884bb75a not found: ID does not exist"
Sep 11 15:07:41.726144 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.726103    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"]
Sep 11 15:07:41.733111 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:41.733074    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/auth-test-model-928943-kserve-97946c58c-2sffs"]
Sep 11 15:07:42.181007 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:07:42.180962    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab9ca370-a41f-418d-9b7e-3800b6349cc8" path="/var/lib/kubelet/pods/ab9ca370-a41f-418d-9b7e-3800b6349cc8/volumes"
Sep 11 15:08:05.179738 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.179699    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"]
Sep 11 15:08:05.180382 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.180226    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7515f258-5922-4549-8c51-f7431b65ad29" containerName="test-curl-7674-64ea67"
Sep 11 15:08:05.180382 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.180248    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7515f258-5922-4549-8c51-f7431b65ad29" containerName="test-curl-7674-64ea67"
Sep 11 15:08:05.180382 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.180266    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy"
Sep 11 15:08:05.180382 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.180276    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy"
Sep 11 15:08:05.180382 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.180296    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab9ca370-a41f-418d-9b7e-3800b6349cc8" containerName="main"
Sep 11 15:08:05.180382 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.180305    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab9ca370-a41f-418d-9b7e-3800b6349cc8" containerName="main"
Sep 11 15:08:05.180679 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.180422    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab9ca370-a41f-418d-9b7e-3800b6349cc8" containerName="main"
Sep 11 15:08:05.180679 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.180435    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy"
Sep 11 15:08:05.180679 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.180442    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7515f258-5922-4549-8c51-f7431b65ad29" containerName="test-curl-7674-64ea67"
Sep 11 15:08:05.180679 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.180448    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy"
Sep 11 15:08:05.183488 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.183446    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.186269 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.186243    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"openshift-service-ca.crt\""
Sep 11 15:08:05.187361 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.187334    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"kube-root-ca.crt\""
Sep 11 15:08:05.187361 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.187349    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"sub-test-model-928943-kserve-self-signed-certs\""
Sep 11 15:08:05.187572 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.187383    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"default-dockercfg-t992d\""
Sep 11 15:08:05.194859 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.194833    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"]
Sep 11 15:08:05.214162 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.214082    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:08:05.285502 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.285442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczz4\" (UniqueName: \"kubernetes.io/projected/9dc5e266-700a-40b0-af7d-f279243d3a21-kube-api-access-qczz4\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.285729 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.285530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-tmp-dir\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.285729 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.285561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-dshm\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.285729 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.285672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-model-cache\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.285907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.285731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc5e266-700a-40b0-af7d-f279243d3a21-tls-certs\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.285907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.285778    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-home\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.386546 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.386508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczz4\" (UniqueName: \"kubernetes.io/projected/9dc5e266-700a-40b0-af7d-f279243d3a21-kube-api-access-qczz4\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.386741 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.386559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-tmp-dir\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.386741 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.386586    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-dshm\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.386741 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.386614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-model-cache\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.386741 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.386641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc5e266-700a-40b0-af7d-f279243d3a21-tls-certs\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.386741 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.386661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-home\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.387094 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.387072    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-model-cache\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.387153 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.387106    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-home\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.387153 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.387138    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-tmp-dir\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.388907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.388881    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-dshm\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.389245 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.389226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc5e266-700a-40b0-af7d-f279243d3a21-tls-certs\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.395788 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.395761    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczz4\" (UniqueName: \"kubernetes.io/projected/9dc5e266-700a-40b0-af7d-f279243d3a21-kube-api-access-qczz4\") pod \"sub-test-model-928943-kserve-7f7b875d45-2lk97\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") " pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.493644 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.493604    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:05.771047 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.771016    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"]
Sep 11 15:08:05.773429 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:08:05.773399    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dc5e266_700a_40b0_af7d_f279243d3a21.slice/crio-777228b9ddcfc206d68ffd16385354d2033d8cdcdd9bb31308776e3fd46e3f55 WatchSource:0}: Error finding container 777228b9ddcfc206d68ffd16385354d2033d8cdcdd9bb31308776e3fd46e3f55: Status 404 returned error can't find the container with id 777228b9ddcfc206d68ffd16385354d2033d8cdcdd9bb31308776e3fd46e3f55
Sep 11 15:08:05.818973 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:05.818943    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97" event={"ID":"9dc5e266-700a-40b0-af7d-f279243d3a21","Type":"ContainerStarted","Data":"777228b9ddcfc206d68ffd16385354d2033d8cdcdd9bb31308776e3fd46e3f55"}
Sep 11 15:08:06.825699 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:06.825656    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97" event={"ID":"9dc5e266-700a-40b0-af7d-f279243d3a21","Type":"ContainerStarted","Data":"4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc"}
Sep 11 15:08:06.849267 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:06.849207    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97" podStartSLOduration=1.84918959 podStartE2EDuration="1.84918959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:06.847949936 +0000 UTC m=+1693.170578305" watchObservedRunningTime="2026-09-11 15:08:06.84918959 +0000 UTC m=+1693.171817960"
Sep 11 15:08:08.628279 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:08.628238    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:08:15.494436 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:15.494392    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:15.494890 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:15.494502    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:15.506970 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:15.506941    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:15.880481 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:15.880387    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:08:16.083003 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:16.082968    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:08:35.619346 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:35.619310    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:08:50.629136 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:50.629100    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:08:54.462983 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:54.462936    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7674-62c777"]
Sep 11 15:08:54.463561 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:54.463538    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy"
Sep 11 15:08:54.463653 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:54.463564    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c2c16e-25b6-4584-9190-1fd68da51f25" containerName="istio-proxy"
Sep 11 15:08:54.465760 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:54.465737    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-62c777"
Sep 11 15:08:54.468713 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:54.468685    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5lj77\""
Sep 11 15:08:54.475739 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:54.475716    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-62c777"]
Sep 11 15:08:54.566057 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:54.566019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxjz9\" (UniqueName: \"kubernetes.io/projected/a4827acc-a54c-4158-ac8f-b7e0a91f3ef0-kube-api-access-xxjz9\") pod \"test-curl-7674-62c777\" (UID: \"a4827acc-a54c-4158-ac8f-b7e0a91f3ef0\") " pod="opendatahub/test-curl-7674-62c777"
Sep 11 15:08:54.667187 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:54.667149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxjz9\" (UniqueName: \"kubernetes.io/projected/a4827acc-a54c-4158-ac8f-b7e0a91f3ef0-kube-api-access-xxjz9\") pod \"test-curl-7674-62c777\" (UID: \"a4827acc-a54c-4158-ac8f-b7e0a91f3ef0\") " pod="opendatahub/test-curl-7674-62c777"
Sep 11 15:08:54.675840 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:54.675805    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxjz9\" (UniqueName: \"kubernetes.io/projected/a4827acc-a54c-4158-ac8f-b7e0a91f3ef0-kube-api-access-xxjz9\") pod \"test-curl-7674-62c777\" (UID: \"a4827acc-a54c-4158-ac8f-b7e0a91f3ef0\") " pod="opendatahub/test-curl-7674-62c777"
Sep 11 15:08:54.776488 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:54.776437    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-62c777"
Sep 11 15:08:55.045998 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:55.045971    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-62c777"]
Sep 11 15:08:55.048255 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:08:55.048222    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4827acc_a54c_4158_ac8f_b7e0a91f3ef0.slice/crio-9c6439d5d78c9b3775966b852ee3bebe39851e2dd8ff6f94b54030c71e5e33a3 WatchSource:0}: Error finding container 9c6439d5d78c9b3775966b852ee3bebe39851e2dd8ff6f94b54030c71e5e33a3: Status 404 returned error can't find the container with id 9c6439d5d78c9b3775966b852ee3bebe39851e2dd8ff6f94b54030c71e5e33a3
Sep 11 15:08:55.068833 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:55.068807    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-62c777" event={"ID":"a4827acc-a54c-4158-ac8f-b7e0a91f3ef0","Type":"ContainerStarted","Data":"9c6439d5d78c9b3775966b852ee3bebe39851e2dd8ff6f94b54030c71e5e33a3"}
Sep 11 15:08:56.075196 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:56.075158    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-62c777" event={"ID":"a4827acc-a54c-4158-ac8f-b7e0a91f3ef0","Type":"ContainerStarted","Data":"8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7"}
Sep 11 15:08:56.093930 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:56.093874    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7674-62c777" podStartSLOduration=2.093830896 podStartE2EDuration="2.093830896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:56.092656319 +0000 UTC m=+1742.415284690" watchObservedRunningTime="2026-09-11 15:08:56.093830896 +0000 UTC m=+1742.416459266"
Sep 11 15:08:56.723816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:56.723775    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7674-62c777"]
Sep 11 15:08:56.728967 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:56.728931    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7674-62c777"]
Sep 11 15:08:56.854758 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:56.854668    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7674-592cca"]
Sep 11 15:08:56.855372 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:56.855347    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4827acc-a54c-4158-ac8f-b7e0a91f3ef0" containerName="test-curl-7674-62c777"
Sep 11 15:08:56.855420 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:56.855378    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4827acc-a54c-4158-ac8f-b7e0a91f3ef0" containerName="test-curl-7674-62c777"
Sep 11 15:08:56.855581 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:56.855564    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4827acc-a54c-4158-ac8f-b7e0a91f3ef0" containerName="test-curl-7674-62c777"
Sep 11 15:08:56.858697 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:56.858682    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-592cca"
Sep 11 15:08:56.862356 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:56.862328    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-592cca"]
Sep 11 15:08:56.988289 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:56.988251    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnltz\" (UniqueName: \"kubernetes.io/projected/02019847-118f-4ef1-b26a-fcfed754317d-kube-api-access-dnltz\") pod \"test-curl-7674-592cca\" (UID: \"02019847-118f-4ef1-b26a-fcfed754317d\") " pod="opendatahub/test-curl-7674-592cca"
Sep 11 15:08:57.020283 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:57.020243    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:08:57.079820 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:08:57.079784    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7674-62c777" container="test-curl-7674-62c777"
Sep 11 15:08:57.089033 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:57.089003    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnltz\" (UniqueName: \"kubernetes.io/projected/02019847-118f-4ef1-b26a-fcfed754317d-kube-api-access-dnltz\") pod \"test-curl-7674-592cca\" (UID: \"02019847-118f-4ef1-b26a-fcfed754317d\") " pod="opendatahub/test-curl-7674-592cca"
Sep 11 15:08:57.098237 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:57.098208    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnltz\" (UniqueName: \"kubernetes.io/projected/02019847-118f-4ef1-b26a-fcfed754317d-kube-api-access-dnltz\") pod \"test-curl-7674-592cca\" (UID: \"02019847-118f-4ef1-b26a-fcfed754317d\") " pod="opendatahub/test-curl-7674-592cca"
Sep 11 15:08:57.170483 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:57.170370    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-592cca"
Sep 11 15:08:57.445069 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:57.445040    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-592cca"]
Sep 11 15:08:57.447024 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:08:57.446999    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02019847_118f_4ef1_b26a_fcfed754317d.slice/crio-74c89594b23c68e4ede87e834c7cbecfc532125521f079514e1c8e185dc69617 WatchSource:0}: Error finding container 74c89594b23c68e4ede87e834c7cbecfc532125521f079514e1c8e185dc69617: Status 404 returned error can't find the container with id 74c89594b23c68e4ede87e834c7cbecfc532125521f079514e1c8e185dc69617
Sep 11 15:08:58.086679 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:58.086643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-592cca" event={"ID":"02019847-118f-4ef1-b26a-fcfed754317d","Type":"ContainerStarted","Data":"36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea"}
Sep 11 15:08:58.086679 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:58.086687    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-592cca" event={"ID":"02019847-118f-4ef1-b26a-fcfed754317d","Type":"ContainerStarted","Data":"74c89594b23c68e4ede87e834c7cbecfc532125521f079514e1c8e185dc69617"}
Sep 11 15:08:58.087245 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:58.086683    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7674-62c777" podUID="a4827acc-a54c-4158-ac8f-b7e0a91f3ef0" containerName="test-curl-7674-62c777" containerID="cri-o://8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7" gracePeriod=2
Sep 11 15:08:58.089150 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:08:58.089115    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-62c777\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="a4827acc-a54c-4158-ac8f-b7e0a91f3ef0" pod="opendatahub/test-curl-7674-62c777"
Sep 11 15:08:58.104220 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:58.104137    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7674-592cca" podStartSLOduration=2.104114919 podStartE2EDuration="2.104114919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:58.102884641 +0000 UTC m=+1744.425513011" watchObservedRunningTime="2026-09-11 15:08:58.104114919 +0000 UTC m=+1744.426743290"
Sep 11 15:08:58.105271 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:08:58.105242    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-62c777\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="a4827acc-a54c-4158-ac8f-b7e0a91f3ef0" pod="opendatahub/test-curl-7674-62c777"
Sep 11 15:08:58.722798 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:58.722759    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7674-592cca"]
Sep 11 15:08:58.727922 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:08:58.727889    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7674-592cca"]
Sep 11 15:08:59.090975 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:08:59.090945    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7674-592cca" container="test-curl-7674-592cca"
Sep 11 15:09:00.096237 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:00.096200    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7674-592cca" podUID="02019847-118f-4ef1-b26a-fcfed754317d" containerName="test-curl-7674-592cca" containerID="cri-o://36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea" gracePeriod=2
Sep 11 15:09:00.098700 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:09:00.098668    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-592cca\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="02019847-118f-4ef1-b26a-fcfed754317d" pod="opendatahub/test-curl-7674-592cca"
Sep 11 15:09:00.279646 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:00.279621    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-62c777"
Sep 11 15:09:00.425179 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:00.425087    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxjz9\" (UniqueName: \"kubernetes.io/projected/a4827acc-a54c-4158-ac8f-b7e0a91f3ef0-kube-api-access-xxjz9\") pod \"a4827acc-a54c-4158-ac8f-b7e0a91f3ef0\" (UID: \"a4827acc-a54c-4158-ac8f-b7e0a91f3ef0\") "
Sep 11 15:09:00.427151 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:00.427119    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4827acc-a54c-4158-ac8f-b7e0a91f3ef0-kube-api-access-xxjz9" (OuterVolumeSpecName: "kube-api-access-xxjz9") pod "a4827acc-a54c-4158-ac8f-b7e0a91f3ef0" (UID: "a4827acc-a54c-4158-ac8f-b7e0a91f3ef0"). InnerVolumeSpecName "kube-api-access-xxjz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:09:00.526036 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:00.525998    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxjz9\" (UniqueName: \"kubernetes.io/projected/a4827acc-a54c-4158-ac8f-b7e0a91f3ef0-kube-api-access-xxjz9\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:09:00.888742 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:00.888701    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:09:01.101771 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:01.101737    2559 generic.go:356] "Generic (PLEG): container finished" podID="a4827acc-a54c-4158-ac8f-b7e0a91f3ef0" containerID="8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7" exitCode=137
Sep 11 15:09:01.102171 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:01.101791    2559 scope.go:117] "RemoveContainer" containerID="8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7"
Sep 11 15:09:01.102171 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:01.101787    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-62c777"
Sep 11 15:09:01.111101 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:01.111080    2559 scope.go:117] "RemoveContainer" containerID="8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7"
Sep 11 15:09:01.111392 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:09:01.111372    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7\": container with ID starting with 8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7 not found: ID does not exist" containerID="8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7"
Sep 11 15:09:01.111571 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:01.111405    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7"} err="failed to get container status \"8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7\": rpc error: code = NotFound desc = could not find container \"8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7\": container with ID starting with 8389d074e8d78ac6a9fb43e758a48cb1d2044abc3ed04a4f35205ab89546bec7 not found: ID does not exist"
Sep 11 15:09:01.113276 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:09:01.113250    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-62c777\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="a4827acc-a54c-4158-ac8f-b7e0a91f3ef0" pod="opendatahub/test-curl-7674-62c777"
Sep 11 15:09:01.854132 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:01.854095    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"]
Sep 11 15:09:01.856946 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:09:01.856895    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-62c777\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="a4827acc-a54c-4158-ac8f-b7e0a91f3ef0" pod="opendatahub/test-curl-7674-62c777"
Sep 11 15:09:01.871902 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:01.871860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:09:02.180765 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:02.180733    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4827acc-a54c-4158-ac8f-b7e0a91f3ef0" path="/var/lib/kubelet/pods/a4827acc-a54c-4158-ac8f-b7e0a91f3ef0/volumes"
Sep 11 15:09:02.289794 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:02.289766    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-592cca"
Sep 11 15:09:02.445388 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:02.445294    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnltz\" (UniqueName: \"kubernetes.io/projected/02019847-118f-4ef1-b26a-fcfed754317d-kube-api-access-dnltz\") pod \"02019847-118f-4ef1-b26a-fcfed754317d\" (UID: \"02019847-118f-4ef1-b26a-fcfed754317d\") "
Sep 11 15:09:02.447407 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:02.447381    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02019847-118f-4ef1-b26a-fcfed754317d-kube-api-access-dnltz" (OuterVolumeSpecName: "kube-api-access-dnltz") pod "02019847-118f-4ef1-b26a-fcfed754317d" (UID: "02019847-118f-4ef1-b26a-fcfed754317d"). InnerVolumeSpecName "kube-api-access-dnltz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:09:02.546488 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:02.546412    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnltz\" (UniqueName: \"kubernetes.io/projected/02019847-118f-4ef1-b26a-fcfed754317d-kube-api-access-dnltz\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:09:03.114840 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:03.114800    2559 generic.go:356] "Generic (PLEG): container finished" podID="02019847-118f-4ef1-b26a-fcfed754317d" containerID="36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea" exitCode=137
Sep 11 15:09:03.114840 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:03.114840    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-592cca"
Sep 11 15:09:03.115065 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:03.114904    2559 scope.go:117] "RemoveContainer" containerID="36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea"
Sep 11 15:09:03.123655 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:03.123637    2559 scope.go:117] "RemoveContainer" containerID="36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea"
Sep 11 15:09:03.123960 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:09:03.123941    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea\": container with ID starting with 36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea not found: ID does not exist" containerID="36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea"
Sep 11 15:09:03.124011 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:03.123969    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea"} err="failed to get container status \"36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea\": rpc error: code = NotFound desc = could not find container \"36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea\": container with ID starting with 36d1a28f2656a4e128216201d6c687a127e594790d96bc4370fdd991522cfbea not found: ID does not exist"
Sep 11 15:09:03.126222 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:09:03.126197    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-592cca\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="02019847-118f-4ef1-b26a-fcfed754317d" pod="opendatahub/test-curl-7674-592cca"
Sep 11 15:09:04.178728 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:09:04.178677    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-592cca\" is forbidden: User \"system:node:ip-10-0-128-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-223.ec2.internal' and this object" podUID="02019847-118f-4ef1-b26a-fcfed754317d" pod="opendatahub/test-curl-7674-592cca"
Sep 11 15:09:04.180185 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:04.180161    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02019847-118f-4ef1-b26a-fcfed754317d" path="/var/lib/kubelet/pods/02019847-118f-4ef1-b26a-fcfed754317d/volumes"
Sep 11 15:09:05.031520 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.031479    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:09:05.795062 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.795021    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"]
Sep 11 15:09:05.795544 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.795434    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02019847-118f-4ef1-b26a-fcfed754317d" containerName="test-curl-7674-592cca"
Sep 11 15:09:05.795544 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.795448    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="02019847-118f-4ef1-b26a-fcfed754317d" containerName="test-curl-7674-592cca"
Sep 11 15:09:05.795544 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.795521    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="02019847-118f-4ef1-b26a-fcfed754317d" containerName="test-curl-7674-592cca"
Sep 11 15:09:05.799845 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.799813    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.803696 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.803664    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-48898f\"/\"default-dockercfg-xslvr\""
Sep 11 15:09:05.803984 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.803964    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-48898f\"/\"rate-test-model-48898f-kserve-self-signed-certs\""
Sep 11 15:09:05.804055 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.803995    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-48898f\"/\"openshift-service-ca.crt\""
Sep 11 15:09:05.804098 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.804040    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-48898f\"/\"kube-root-ca.crt\""
Sep 11 15:09:05.811021 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.810993    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"]
Sep 11 15:09:05.875733 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.875691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:09:05.883600 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.883568    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48acaa45-b373-4138-bef5-c386037bd7e6-tls-certs\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.883802 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.883623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-dshm\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.883802 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.883644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjr7x\" (UniqueName: \"kubernetes.io/projected/48acaa45-b373-4138-bef5-c386037bd7e6-kube-api-access-wjr7x\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.883802 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.883707    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-model-cache\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.883802 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.883751    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-tmp-dir\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.883802 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.883796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-home\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.985249 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.985205    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48acaa45-b373-4138-bef5-c386037bd7e6-tls-certs\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.985440 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.985352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-dshm\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.985440 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.985372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjr7x\" (UniqueName: \"kubernetes.io/projected/48acaa45-b373-4138-bef5-c386037bd7e6-kube-api-access-wjr7x\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.985440 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.985394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-model-cache\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.985440 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.985414    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-tmp-dir\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.985693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.985443    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-home\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.985907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.985875    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-tmp-dir\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.986044 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.985966    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-model-cache\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.986044 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.985992    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-home\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.987719 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.987698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-dshm\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.987846 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.987830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48acaa45-b373-4138-bef5-c386037bd7e6-tls-certs\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:05.996254 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:05.996226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjr7x\" (UniqueName: \"kubernetes.io/projected/48acaa45-b373-4138-bef5-c386037bd7e6-kube-api-access-wjr7x\") pod \"rate-test-model-48898f-kserve-76944b8c87-4mjhw\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") " pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:06.110626 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:06.110529    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:06.396318 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:06.396255    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"]
Sep 11 15:09:06.398669 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:09:06.398643    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48acaa45_b373_4138_bef5_c386037bd7e6.slice/crio-0d559d94f398e0c781abd81b07e605482ad95d6bbc9697c35d81051188600f4b WatchSource:0}: Error finding container 0d559d94f398e0c781abd81b07e605482ad95d6bbc9697c35d81051188600f4b: Status 404 returned error can't find the container with id 0d559d94f398e0c781abd81b07e605482ad95d6bbc9697c35d81051188600f4b
Sep 11 15:09:07.138432 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:07.138395    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw" event={"ID":"48acaa45-b373-4138-bef5-c386037bd7e6","Type":"ContainerStarted","Data":"6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835"}
Sep 11 15:09:07.138872 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:07.138442    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw" event={"ID":"48acaa45-b373-4138-bef5-c386037bd7e6","Type":"ContainerStarted","Data":"0d559d94f398e0c781abd81b07e605482ad95d6bbc9697c35d81051188600f4b"}
Sep 11 15:09:07.158808 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:07.158741    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw" podStartSLOduration=2.158719031 podStartE2EDuration="2.158719031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:07.157666495 +0000 UTC m=+1753.480294868" watchObservedRunningTime="2026-09-11 15:09:07.158719031 +0000 UTC m=+1753.481347403"
Sep 11 15:09:15.520301 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:15.520263    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:09:16.111335 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:16.111292    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:16.111534 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:16.111462    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:16.123746 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:16.123714    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:16.198351 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:16.198315    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:09:16.873443 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:16.873401    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97" podUID="9dc5e266-700a-40b0-af7d-f279243d3a21" containerName="main" containerID="cri-o://4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc" gracePeriod=45
Sep 11 15:09:17.120545 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.120518    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:09:17.193081 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.192972    2559 generic.go:356] "Generic (PLEG): container finished" podID="9dc5e266-700a-40b0-af7d-f279243d3a21" containerID="4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc" exitCode=0
Sep 11 15:09:17.193081 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.193021    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"
Sep 11 15:09:17.193081 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.193074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97" event={"ID":"9dc5e266-700a-40b0-af7d-f279243d3a21","Type":"ContainerDied","Data":"4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc"}
Sep 11 15:09:17.193406 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.193115    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97" event={"ID":"9dc5e266-700a-40b0-af7d-f279243d3a21","Type":"ContainerDied","Data":"777228b9ddcfc206d68ffd16385354d2033d8cdcdd9bb31308776e3fd46e3f55"}
Sep 11 15:09:17.193406 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.193139    2559 scope.go:117] "RemoveContainer" containerID="4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc"
Sep 11 15:09:17.200671 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.200649    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-tmp-dir\") pod \"9dc5e266-700a-40b0-af7d-f279243d3a21\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") "
Sep 11 15:09:17.200774 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.200698    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qczz4\" (UniqueName: \"kubernetes.io/projected/9dc5e266-700a-40b0-af7d-f279243d3a21-kube-api-access-qczz4\") pod \"9dc5e266-700a-40b0-af7d-f279243d3a21\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") "
Sep 11 15:09:17.200774 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.200739    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-home\") pod \"9dc5e266-700a-40b0-af7d-f279243d3a21\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") "
Sep 11 15:09:17.200882 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.200839    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc5e266-700a-40b0-af7d-f279243d3a21-tls-certs\") pod \"9dc5e266-700a-40b0-af7d-f279243d3a21\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") "
Sep 11 15:09:17.200949 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.200879    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-dshm\") pod \"9dc5e266-700a-40b0-af7d-f279243d3a21\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") "
Sep 11 15:09:17.200949 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.200939    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-model-cache\") pod \"9dc5e266-700a-40b0-af7d-f279243d3a21\" (UID: \"9dc5e266-700a-40b0-af7d-f279243d3a21\") "
Sep 11 15:09:17.201052 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.201025    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9dc5e266-700a-40b0-af7d-f279243d3a21" (UID: "9dc5e266-700a-40b0-af7d-f279243d3a21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:09:17.201108 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.201042    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-home" (OuterVolumeSpecName: "home") pod "9dc5e266-700a-40b0-af7d-f279243d3a21" (UID: "9dc5e266-700a-40b0-af7d-f279243d3a21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:09:17.201235 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.201217    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-model-cache" (OuterVolumeSpecName: "model-cache") pod "9dc5e266-700a-40b0-af7d-f279243d3a21" (UID: "9dc5e266-700a-40b0-af7d-f279243d3a21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:09:17.201312 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.201294    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:09:17.201355 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.201314    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:09:17.201355 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.201325    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:09:17.202384 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.202361    2559 scope.go:117] "RemoveContainer" containerID="4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc"
Sep 11 15:09:17.202747 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:09:17.202724    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc\": container with ID starting with 4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc not found: ID does not exist" containerID="4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc"
Sep 11 15:09:17.202834 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.202760    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc"} err="failed to get container status \"4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc\": rpc error: code = NotFound desc = could not find container \"4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc\": container with ID starting with 4f1a624fecb5af5f6a4b2b3d9d89082fe8964ac4e697cb966481dcbdc96678fc not found: ID does not exist"
Sep 11 15:09:17.203056 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.203032    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-dshm" (OuterVolumeSpecName: "dshm") pod "9dc5e266-700a-40b0-af7d-f279243d3a21" (UID: "9dc5e266-700a-40b0-af7d-f279243d3a21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:09:17.203127 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.203064    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc5e266-700a-40b0-af7d-f279243d3a21-kube-api-access-qczz4" (OuterVolumeSpecName: "kube-api-access-qczz4") pod "9dc5e266-700a-40b0-af7d-f279243d3a21" (UID: "9dc5e266-700a-40b0-af7d-f279243d3a21"). InnerVolumeSpecName "kube-api-access-qczz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:09:17.203204 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.203185    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc5e266-700a-40b0-af7d-f279243d3a21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9dc5e266-700a-40b0-af7d-f279243d3a21" (UID: "9dc5e266-700a-40b0-af7d-f279243d3a21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:09:17.302807 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.302768    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qczz4\" (UniqueName: \"kubernetes.io/projected/9dc5e266-700a-40b0-af7d-f279243d3a21-kube-api-access-qczz4\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:09:17.302807 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.302801    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc5e266-700a-40b0-af7d-f279243d3a21-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:09:17.302807 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.302811    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc5e266-700a-40b0-af7d-f279243d3a21-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:09:17.517203 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.517168    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"]
Sep 11 15:09:17.521118 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:17.521089    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/sub-test-model-928943-kserve-7f7b875d45-2lk97"]
Sep 11 15:09:18.180182 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:18.180147    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc5e266-700a-40b0-af7d-f279243d3a21" path="/var/lib/kubelet/pods/9dc5e266-700a-40b0-af7d-f279243d3a21/volumes"
Sep 11 15:09:29.212043 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:29.212006    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:09:29.267786 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:29.267755    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:09:33.248866 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:33.248821    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:09:34.524530 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:34.524476    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:09:38.121637 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:38.121603    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:09:54.727065 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:09:54.727027    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:10:03.916341 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:03.916299    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:10:09.087350 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:09.087316    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"]
Sep 11 15:10:09.114541 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:09.114501    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:10.111629 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:10.111590    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:10.556390 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:10.556352    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:10.864978 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:10.864891    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:11.004345 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:11.004299    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:12.687621 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:12.687589    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:13.530244 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:13.530196    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:14.157831 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:14.157793    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:14.255319 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:14.255282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:14.449198 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:14.449099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:14.830357 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:14.830319    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:10:15.247065 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:15.247008    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:16.211796 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:16.211760    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:16.277555 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:16.277517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:16.452265 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:16.452211    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:17.354091 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:17.354052    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:17.454507 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:17.454440    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:24.106388 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.106348    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw" podUID="48acaa45-b373-4138-bef5-c386037bd7e6" containerName="main" containerID="cri-o://6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835" gracePeriod=45
Sep 11 15:10:24.346350 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.346322    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:10:24.351960 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.351937    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-tmp-dir\") pod \"48acaa45-b373-4138-bef5-c386037bd7e6\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") "
Sep 11 15:10:24.352102 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.351984    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-model-cache\") pod \"48acaa45-b373-4138-bef5-c386037bd7e6\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") "
Sep 11 15:10:24.352102 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.352014    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48acaa45-b373-4138-bef5-c386037bd7e6-tls-certs\") pod \"48acaa45-b373-4138-bef5-c386037bd7e6\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") "
Sep 11 15:10:24.352102 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.352053    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjr7x\" (UniqueName: \"kubernetes.io/projected/48acaa45-b373-4138-bef5-c386037bd7e6-kube-api-access-wjr7x\") pod \"48acaa45-b373-4138-bef5-c386037bd7e6\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") "
Sep 11 15:10:24.352102 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.352083    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-dshm\") pod \"48acaa45-b373-4138-bef5-c386037bd7e6\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") "
Sep 11 15:10:24.352392 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.352116    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-home\") pod \"48acaa45-b373-4138-bef5-c386037bd7e6\" (UID: \"48acaa45-b373-4138-bef5-c386037bd7e6\") "
Sep 11 15:10:24.352392 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.352130    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48acaa45-b373-4138-bef5-c386037bd7e6" (UID: "48acaa45-b373-4138-bef5-c386037bd7e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:10:24.352392 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.352266    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "48acaa45-b373-4138-bef5-c386037bd7e6" (UID: "48acaa45-b373-4138-bef5-c386037bd7e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:10:24.352574 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.352429    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-home" (OuterVolumeSpecName: "home") pod "48acaa45-b373-4138-bef5-c386037bd7e6" (UID: "48acaa45-b373-4138-bef5-c386037bd7e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:10:24.352574 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.352566    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:10:24.352668 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.352588    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:10:24.352668 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.352601    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:10:24.354248 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.354225    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48acaa45-b373-4138-bef5-c386037bd7e6-kube-api-access-wjr7x" (OuterVolumeSpecName: "kube-api-access-wjr7x") pod "48acaa45-b373-4138-bef5-c386037bd7e6" (UID: "48acaa45-b373-4138-bef5-c386037bd7e6"). InnerVolumeSpecName "kube-api-access-wjr7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:10:24.354248 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.354232    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-dshm" (OuterVolumeSpecName: "dshm") pod "48acaa45-b373-4138-bef5-c386037bd7e6" (UID: "48acaa45-b373-4138-bef5-c386037bd7e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:10:24.354248 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.354244    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48acaa45-b373-4138-bef5-c386037bd7e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48acaa45-b373-4138-bef5-c386037bd7e6" (UID: "48acaa45-b373-4138-bef5-c386037bd7e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:10:24.453397 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.453361    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48acaa45-b373-4138-bef5-c386037bd7e6-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:10:24.453397 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.453392    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48acaa45-b373-4138-bef5-c386037bd7e6-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:10:24.453397 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.453406    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjr7x\" (UniqueName: \"kubernetes.io/projected/48acaa45-b373-4138-bef5-c386037bd7e6-kube-api-access-wjr7x\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:10:24.522792 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.522758    2559 generic.go:356] "Generic (PLEG): container finished" podID="48acaa45-b373-4138-bef5-c386037bd7e6" containerID="6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835" exitCode=0
Sep 11 15:10:24.522982 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.522802    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"
Sep 11 15:10:24.522982 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.522842    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw" event={"ID":"48acaa45-b373-4138-bef5-c386037bd7e6","Type":"ContainerDied","Data":"6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835"}
Sep 11 15:10:24.522982 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.522884    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw" event={"ID":"48acaa45-b373-4138-bef5-c386037bd7e6","Type":"ContainerDied","Data":"0d559d94f398e0c781abd81b07e605482ad95d6bbc9697c35d81051188600f4b"}
Sep 11 15:10:24.522982 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.522902    2559 scope.go:117] "RemoveContainer" containerID="6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835"
Sep 11 15:10:24.532031 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.532014    2559 scope.go:117] "RemoveContainer" containerID="6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835"
Sep 11 15:10:24.532325 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:10:24.532308    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835\": container with ID starting with 6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835 not found: ID does not exist" containerID="6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835"
Sep 11 15:10:24.532373 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.532333    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835"} err="failed to get container status \"6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835\": rpc error: code = NotFound desc = could not find container \"6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835\": container with ID starting with 6495e4c1c3671558cd292b560e63bf2b300d9f0baf3850057bc58eb962c93835 not found: ID does not exist"
Sep 11 15:10:24.548719 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.548680    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"]
Sep 11 15:10:24.556413 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:24.556384    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/rate-test-model-48898f-kserve-76944b8c87-4mjhw"]
Sep 11 15:10:25.319060 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:25.319018    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:10:26.181875 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:26.181837    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48acaa45-b373-4138-bef5-c386037bd7e6" path="/var/lib/kubelet/pods/48acaa45-b373-4138-bef5-c386037bd7e6/volumes"
Sep 11 15:10:26.961280 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:26.961234    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:38.532622 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:38.532586    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:39.287803 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:39.287766    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:39.426758 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:39.426719    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:39.981109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:39.981078    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:42.581960 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:42.581922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:42.702331 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:42.702298    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:43.389898 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:43.389860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:43.626977 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:43.626941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:44.355368 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:44.355334    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:45.156936 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:45.156898    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:46.076441 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.076406    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"]
Sep 11 15:10:46.076962 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.076938    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc5e266-700a-40b0-af7d-f279243d3a21" containerName="main"
Sep 11 15:10:46.076962 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.076959    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc5e266-700a-40b0-af7d-f279243d3a21" containerName="main"
Sep 11 15:10:46.077165 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.076970    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48acaa45-b373-4138-bef5-c386037bd7e6" containerName="main"
Sep 11 15:10:46.077165 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.076976    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="48acaa45-b373-4138-bef5-c386037bd7e6" containerName="main"
Sep 11 15:10:46.077165 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.077101    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="48acaa45-b373-4138-bef5-c386037bd7e6" containerName="main"
Sep 11 15:10:46.077165 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.077115    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc5e266-700a-40b0-af7d-f279243d3a21" containerName="main"
Sep 11 15:10:46.080190 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.080171    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.083018 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.082991    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4ee29a-serving-cert\""
Sep 11 15:10:46.093036 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.093005    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"]
Sep 11 15:10:46.109946 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.109911    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:46.144591 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.144557    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f873ce-5342-471a-85b4-2b7ca9bc9455-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.144802 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.144676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.144802 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.144718    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-metrics-certs\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.144802 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.144743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2s7k\" (UniqueName: \"kubernetes.io/projected/c4f873ce-5342-471a-85b4-2b7ca9bc9455-kube-api-access-x2s7k\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.179083 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.179041    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:46.246224 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.246189    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.246380 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.246234    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-metrics-certs\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.246380 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.246251    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2s7k\" (UniqueName: \"kubernetes.io/projected/c4f873ce-5342-471a-85b4-2b7ca9bc9455-kube-api-access-x2s7k\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.246380 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.246311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f873ce-5342-471a-85b4-2b7ca9bc9455-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.246380 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:10:46.246340    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-serving-cert: secret "maas-api-e2e-ipp-b-4ee29a-serving-cert" not found
Sep 11 15:10:46.246566 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:10:46.246412    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-maas-api-tls podName:c4f873ce-5342-471a-85b4-2b7ca9bc9455 nodeName:}" failed. No retries permitted until 2026-09-11 15:10:46.746391601 +0000 UTC m=+1853.069019949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-maas-api-tls") pod "maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z" (UID: "c4f873ce-5342-471a-85b4-2b7ca9bc9455") : secret "maas-api-e2e-ipp-b-4ee29a-serving-cert" not found
Sep 11 15:10:46.246943 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.246923    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f873ce-5342-471a-85b4-2b7ca9bc9455-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.248660 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.248641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-metrics-certs\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.255719 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.255693    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2s7k\" (UniqueName: \"kubernetes.io/projected/c4f873ce-5342-471a-85b4-2b7ca9bc9455-kube-api-access-x2s7k\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.507949 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.507909    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:46.611721 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.611673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:46.696480 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.696435    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:10:46.750737 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.750709    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.753066 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.753043    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:46.994361 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:46.994328    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:47.272155 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:47.272084    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"]
Sep 11 15:10:47.274869 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:10:47.274836    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f873ce_5342_471a_85b4_2b7ca9bc9455.slice/crio-9fe35c07e45ff507afc759f2b433129e54f0cea5e99e45ca683e8c5624d5af5f WatchSource:0}: Error finding container 9fe35c07e45ff507afc759f2b433129e54f0cea5e99e45ca683e8c5624d5af5f: Status 404 returned error can't find the container with id 9fe35c07e45ff507afc759f2b433129e54f0cea5e99e45ca683e8c5624d5af5f
Sep 11 15:10:47.276248 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:47.276225    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 15:10:47.635188 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:47.635107    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z" event={"ID":"c4f873ce-5342-471a-85b4-2b7ca9bc9455","Type":"ContainerStarted","Data":"9fe35c07e45ff507afc759f2b433129e54f0cea5e99e45ca683e8c5624d5af5f"}
Sep 11 15:10:48.641825 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:48.641777    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z" event={"ID":"c4f873ce-5342-471a-85b4-2b7ca9bc9455","Type":"ContainerStarted","Data":"ee26ab2cb0adadf00a2904cc46eda8c2798d10ed6151b913d6321a1cbcd72880"}
Sep 11 15:10:48.642303 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:48.641865    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:10:48.661684 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:48.661638    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z" podStartSLOduration=2.194590293 podStartE2EDuration="2.661624858s" totalImagesPullingTime="467.034565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:48.660241143 +0000 UTC m=+1854.982869514" watchObservedRunningTime="2026-09-11 15:10:48.661624858 +0000 UTC m=+1854.984253229"
Sep 11 15:10:51.529608 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:51.529577    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:10:54.652434 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:10:54.652398    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:11:08.287310 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:08.287267    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:11:08.390556 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:08.390523    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:11:09.130964 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:09.130915    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:11:09.846095 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:09.846034    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:11:10.584291 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:10.584222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:11:11.407964 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:11.407924    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:11:38.717538 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.717504    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"]
Sep 11 15:11:38.720989 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.720969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.724752 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.724731    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-177c81\"/\"openshift-service-ca.crt\""
Sep 11 15:11:38.724752 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.724744    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-177c81\"/\"default-dockercfg-84spx\""
Sep 11 15:11:38.725690 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.725676    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-177c81\"/\"kube-root-ca.crt\""
Sep 11 15:11:38.725995 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.725978    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-177c81\"/\"ipp-route-177c81-kserve-self-signed-certs\""
Sep 11 15:11:38.734384 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.734360    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"]
Sep 11 15:11:38.760509 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.760481    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:11:38.852834 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.852805    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tls-certs\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.852988 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.852878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-dshm\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.852988 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.852899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tmp-dir\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.852988 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.852918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-home\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.852988 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.852934    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-model-cache\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.852988 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.852952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpw4s\" (UniqueName: \"kubernetes.io/projected/c0a351d5-4cfe-4b87-989b-4d256e2afb71-kube-api-access-fpw4s\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.953895 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.953848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tls-certs\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.954100 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.953976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-dshm\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.954100 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.954007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tmp-dir\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.954100 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.954031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-home\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.954275 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.954097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-model-cache\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.954275 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.954145    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpw4s\" (UniqueName: \"kubernetes.io/projected/c0a351d5-4cfe-4b87-989b-4d256e2afb71-kube-api-access-fpw4s\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.954529 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.954497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-home\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.954529 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.954520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-model-cache\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.954714 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.954631    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tmp-dir\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.956381 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.956346    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-dshm\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.956542 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.956524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tls-certs\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:38.968561 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:38.968496    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpw4s\" (UniqueName: \"kubernetes.io/projected/c0a351d5-4cfe-4b87-989b-4d256e2afb71-kube-api-access-fpw4s\") pod \"ipp-route-177c81-kserve-7489c9bf95-tck26\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") " pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:39.029786 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:39.029744    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:39.311477 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:39.311435    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"]
Sep 11 15:11:39.313217 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:11:39.313191    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0a351d5_4cfe_4b87_989b_4d256e2afb71.slice/crio-445699ea1d5b4f027d28a06a9723e1cb3b9617d170bf79d60dc40e3b5e93f345 WatchSource:0}: Error finding container 445699ea1d5b4f027d28a06a9723e1cb3b9617d170bf79d60dc40e3b5e93f345: Status 404 returned error can't find the container with id 445699ea1d5b4f027d28a06a9723e1cb3b9617d170bf79d60dc40e3b5e93f345
Sep 11 15:11:39.894240 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:39.894195    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26" event={"ID":"c0a351d5-4cfe-4b87-989b-4d256e2afb71","Type":"ContainerStarted","Data":"c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5"}
Sep 11 15:11:39.894240 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:39.894245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26" event={"ID":"c0a351d5-4cfe-4b87-989b-4d256e2afb71","Type":"ContainerStarted","Data":"445699ea1d5b4f027d28a06a9723e1cb3b9617d170bf79d60dc40e3b5e93f345"}
Sep 11 15:11:39.916166 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:39.916115    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26" podStartSLOduration=1.9160988479999999 podStartE2EDuration="1.916098848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:11:39.914930085 +0000 UTC m=+1906.237558455" watchObservedRunningTime="2026-09-11 15:11:39.916098848 +0000 UTC m=+1906.238727229"
Sep 11 15:11:42.840693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:42.840653    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:11:42.853559 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:42.853519    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:11:49.030781 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:49.030722    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:49.030781 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:49.030787    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:49.043578 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:49.043548    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:49.955878 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:49.955844    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:11:59.029703 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:11:59.029664    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:12:05.792548 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:05.792512    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:06.113252 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.113166    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"]
Sep 11 15:12:06.116886 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.116860    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.120535 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.120511    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-e04dee-openshift-default-dockercfg-9lg6q\""
Sep 11 15:12:06.131835 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.131807    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"]
Sep 11 15:12:06.160906 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.160858    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:06.225780 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.225743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-certs\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.225780 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.225780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmc2r\" (UniqueName: \"kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-kube-api-access-nmc2r\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.226018 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.225878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-socket\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.226018 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.225933    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-podinfo\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.226018 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.225953    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16bc768f-a604-4d60-a7e6-17783dd28cbc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.226018 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.226004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-envoy\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.226170 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.226059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-token\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.226170 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.226081    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-credential-socket\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.226170 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.226106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-data\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327036 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.326996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-socket\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327228 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-podinfo\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327228 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16bc768f-a604-4d60-a7e6-17783dd28cbc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327228 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-envoy\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327228 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-token\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327228 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-credential-socket\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327228 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-data\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327623 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-certs\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327623 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-socket\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327623 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327565    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmc2r\" (UniqueName: \"kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-kube-api-access-nmc2r\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327623 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327547    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-certs\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327828 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327619    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-credential-socket\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.327828 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.327654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-data\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.328181 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.328162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16bc768f-a604-4d60-a7e6-17783dd28cbc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.329665 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.329644    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-envoy\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.329979 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.329958    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-podinfo\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.337558 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.337535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-token\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.337751 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.337727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmc2r\" (UniqueName: \"kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-kube-api-access-nmc2r\") pod \"e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") " pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.429663 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.429581    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:06.707251 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.707167    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:06.708997 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.708972    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"]
Sep 11 15:12:06.711022 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:12:06.710999    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bc768f_a604_4d60_a7e6_17783dd28cbc.slice/crio-dd32df4dffa621cae0f92ef80ba0848e2d9f9368f53c1604460daa064593350b WatchSource:0}: Error finding container dd32df4dffa621cae0f92ef80ba0848e2d9f9368f53c1604460daa064593350b: Status 404 returned error can't find the container with id dd32df4dffa621cae0f92ef80ba0848e2d9f9368f53c1604460daa064593350b
Sep 11 15:12:06.713305 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.713274    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:12:06.713397 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.713352    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:12:06.713477 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:06.713398    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:12:07.028142 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:07.028099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" event={"ID":"16bc768f-a604-4d60-a7e6-17783dd28cbc","Type":"ContainerStarted","Data":"d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940"}
Sep 11 15:12:07.028587 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:07.028160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" event={"ID":"16bc768f-a604-4d60-a7e6-17783dd28cbc","Type":"ContainerStarted","Data":"dd32df4dffa621cae0f92ef80ba0848e2d9f9368f53c1604460daa064593350b"}
Sep 11 15:12:07.050444 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:07.050386    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" podStartSLOduration=1.050368132 podStartE2EDuration="1.050368132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:12:07.048773334 +0000 UTC m=+1933.371401717" watchObservedRunningTime="2026-09-11 15:12:07.050368132 +0000 UTC m=+1933.372996501"
Sep 11 15:12:07.430855 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:07.430759    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:07.436001 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:07.435975    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:08.032779 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:08.032742    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:08.033576 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:08.033556    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:08.338025 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:08.337924    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:09.089268 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:09.089231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:09.180774 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:09.180724    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:09.349561 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:09.349436    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:10.187484 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:10.187435    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:11.361874 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:11.361822    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:12.181390 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:12.181342    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:13.016234 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:13.016199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:13.116561 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:13.116521    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:13.522101 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:13.522056    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:13.620192 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:13.620155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:13.720012 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:13.719969    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:18.723171 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:18.723137    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:12:29.138847 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:29.138810    2559 generic.go:356] "Generic (PLEG): container finished" podID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerID="d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940" exitCode=0
Sep 11 15:12:29.139282 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:29.138885    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" event={"ID":"16bc768f-a604-4d60-a7e6-17783dd28cbc","Type":"ContainerDied","Data":"d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940"}
Sep 11 15:12:29.139332 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:29.139286    2559 scope.go:117] "RemoveContainer" containerID="d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940"
Sep 11 15:12:29.140417 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:29.140392    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:12:29.140510 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:29.140482    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:12:29.140553 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:29.140517    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 15:12:30.146497 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:30.146443    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" event={"ID":"16bc768f-a604-4d60-a7e6-17783dd28cbc","Type":"ContainerStarted","Data":"3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352"}
Sep 11 15:12:30.430501 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:30.430410    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:30.430647 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:30.430624    2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Sep 11 15:12:30.430695 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:30.430679    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Sep 11 15:12:31.430763 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:31.430726    2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Sep 11 15:12:31.431176 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:31.430785    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Sep 11 15:12:32.430585 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:32.430545    2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Sep 11 15:12:32.430773 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:32.430610    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Sep 11 15:12:33.430969 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:33.430886    2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Sep 11 15:12:33.431408 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:33.431379    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Sep 11 15:12:34.430720 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:34.430683    2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Sep 11 15:12:34.430907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:34.430740    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Sep 11 15:12:35.104102 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:35.104060    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:35.433821 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:35.433729    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:35.434231 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:35.434204    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:35.434931 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:35.434909    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:12:35.794908 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:35.794880    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:36.648798 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:36.648757    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:37.362917 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:37.362881    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:38.198609 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:38.198570    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:38.877584 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:38.877548    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:39.226234 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:39.226199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:40.695446 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:40.695397    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:41.373510 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:41.373470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:42.068760 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:42.068726    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:42.919399 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:42.919362    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:43.602680 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:43.602632    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:44.317641 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:44.317604    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:50.807927 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:50.807884    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:50.889602 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:50.889548    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:50.985860 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:50.985823    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:51.050312 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:51.050279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:51.477994 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:51.477955    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:51.578728 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:51.578693    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:52.253980 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:52.253933    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:53.118703 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:53.118654    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:53.815584 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:53.815548    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:54.622634 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:54.622596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:55.324143 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:55.324101    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:12:57.523220 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:57.523176    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:57.628607 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:57.628567    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:12:57.788212 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:57.788109    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"]
Sep 11 15:12:57.788420 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:57.788386    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy" containerID="cri-o://3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352" gracePeriod=30
Sep 11 15:12:57.806168 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:12:57.806124    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:01.788018 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:01.787979    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:02.679361 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:02.679317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:03.053521 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.051672    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:13:03.191957 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.191908    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-podinfo\") pod \"16bc768f-a604-4d60-a7e6-17783dd28cbc\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") "
Sep 11 15:13:03.192173 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.191968    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-socket\") pod \"16bc768f-a604-4d60-a7e6-17783dd28cbc\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") "
Sep 11 15:13:03.192173 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192019    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16bc768f-a604-4d60-a7e6-17783dd28cbc-istiod-ca-cert\") pod \"16bc768f-a604-4d60-a7e6-17783dd28cbc\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") "
Sep 11 15:13:03.192173 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192065    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-credential-socket\") pod \"16bc768f-a604-4d60-a7e6-17783dd28cbc\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") "
Sep 11 15:13:03.192173 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192133    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmc2r\" (UniqueName: \"kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-kube-api-access-nmc2r\") pod \"16bc768f-a604-4d60-a7e6-17783dd28cbc\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") "
Sep 11 15:13:03.192173 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192162    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-envoy\") pod \"16bc768f-a604-4d60-a7e6-17783dd28cbc\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") "
Sep 11 15:13:03.192434 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192185    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-data\") pod \"16bc768f-a604-4d60-a7e6-17783dd28cbc\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") "
Sep 11 15:13:03.192434 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192216    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-certs\") pod \"16bc768f-a604-4d60-a7e6-17783dd28cbc\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") "
Sep 11 15:13:03.192434 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192270    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "16bc768f-a604-4d60-a7e6-17783dd28cbc" (UID: "16bc768f-a604-4d60-a7e6-17783dd28cbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:13:03.192434 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192279    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-token\") pod \"16bc768f-a604-4d60-a7e6-17783dd28cbc\" (UID: \"16bc768f-a604-4d60-a7e6-17783dd28cbc\") "
Sep 11 15:13:03.192687 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192550    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "16bc768f-a604-4d60-a7e6-17783dd28cbc" (UID: "16bc768f-a604-4d60-a7e6-17783dd28cbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:13:03.192687 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192586    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16bc768f-a604-4d60-a7e6-17783dd28cbc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "16bc768f-a604-4d60-a7e6-17783dd28cbc" (UID: "16bc768f-a604-4d60-a7e6-17783dd28cbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:13:03.192687 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192656    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-data" (OuterVolumeSpecName: "istio-data") pod "16bc768f-a604-4d60-a7e6-17783dd28cbc" (UID: "16bc768f-a604-4d60-a7e6-17783dd28cbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:13:03.192907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192705    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "16bc768f-a604-4d60-a7e6-17783dd28cbc" (UID: "16bc768f-a604-4d60-a7e6-17783dd28cbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:13:03.192907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192829    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-credential-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:03.192907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192850    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-data\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:03.192907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192863    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:03.192907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192877    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-workload-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:03.192907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.192892    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16bc768f-a604-4d60-a7e6-17783dd28cbc-istiod-ca-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:03.194553 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.194517    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-kube-api-access-nmc2r" (OuterVolumeSpecName: "kube-api-access-nmc2r") pod "16bc768f-a604-4d60-a7e6-17783dd28cbc" (UID: "16bc768f-a604-4d60-a7e6-17783dd28cbc"). InnerVolumeSpecName "kube-api-access-nmc2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:13:03.194553 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.194547    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-token" (OuterVolumeSpecName: "istio-token") pod "16bc768f-a604-4d60-a7e6-17783dd28cbc" (UID: "16bc768f-a604-4d60-a7e6-17783dd28cbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:13:03.194708 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.194559    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "16bc768f-a604-4d60-a7e6-17783dd28cbc" (UID: "16bc768f-a604-4d60-a7e6-17783dd28cbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 15:13:03.194708 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.194598    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "16bc768f-a604-4d60-a7e6-17783dd28cbc" (UID: "16bc768f-a604-4d60-a7e6-17783dd28cbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:13:03.294192 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.294152    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-token\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:03.294192 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.294184    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-podinfo\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:03.294192 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.294193    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmc2r\" (UniqueName: \"kubernetes.io/projected/16bc768f-a604-4d60-a7e6-17783dd28cbc-kube-api-access-nmc2r\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:03.294192 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.294203    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16bc768f-a604-4d60-a7e6-17783dd28cbc-istio-envoy\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:03.318475 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.318413    2559 generic.go:356] "Generic (PLEG): container finished" podID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerID="3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352" exitCode=0
Sep 11 15:13:03.318625 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.318491    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"
Sep 11 15:13:03.318625 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.318508    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" event={"ID":"16bc768f-a604-4d60-a7e6-17783dd28cbc","Type":"ContainerDied","Data":"3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352"}
Sep 11 15:13:03.318625 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.318564    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr" event={"ID":"16bc768f-a604-4d60-a7e6-17783dd28cbc","Type":"ContainerDied","Data":"dd32df4dffa621cae0f92ef80ba0848e2d9f9368f53c1604460daa064593350b"}
Sep 11 15:13:03.318625 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.318592    2559 scope.go:117] "RemoveContainer" containerID="3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352"
Sep 11 15:13:03.328612 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.328591    2559 scope.go:117] "RemoveContainer" containerID="d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940"
Sep 11 15:13:03.337651 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.337629    2559 scope.go:117] "RemoveContainer" containerID="3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352"
Sep 11 15:13:03.337989 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:13:03.337967    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352\": container with ID starting with 3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352 not found: ID does not exist" containerID="3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352"
Sep 11 15:13:03.338077 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.338001    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352"} err="failed to get container status \"3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352\": rpc error: code = NotFound desc = could not find container \"3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352\": container with ID starting with 3296bdab87ea6946891e4d738ade8cebce6775be71963cfb4fe69467dd828352 not found: ID does not exist"
Sep 11 15:13:03.338077 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.338027    2559 scope.go:117] "RemoveContainer" containerID="d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940"
Sep 11 15:13:03.338297 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:13:03.338280    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940\": container with ID starting with d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940 not found: ID does not exist" containerID="d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940"
Sep 11 15:13:03.338343 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.338302    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940"} err="failed to get container status \"d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940\": rpc error: code = NotFound desc = could not find container \"d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940\": container with ID starting with d46339788d65647d5ec92f4f0bf1bf9e6d9640ed2202d07206f1d5b30c620940 not found: ID does not exist"
Sep 11 15:13:03.343177 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.343143    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"]
Sep 11 15:13:03.346139 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.346106    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e04dee-openshift-default-6b5c749998-xp6mr"]
Sep 11 15:13:03.397191 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:03.397153    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:04.180265 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:04.180232    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" path="/var/lib/kubelet/pods/16bc768f-a604-4d60-a7e6-17783dd28cbc/volumes"
Sep 11 15:13:04.213433 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:04.213398    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:04.913691 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:04.913651    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:05.608548 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:05.608515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:14.817869 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:14.817828    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:13:17.366133 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:17.366093    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:17.467468 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:17.467426    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:17.569617 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:17.569577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:17.666777 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:17.666683    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:17.970268 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:17.970231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:18.066652 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:18.066617    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:18.803186 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:18.803146    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:19.666391 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:19.666347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:20.510640 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:20.510600    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:22.937411 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:22.937372    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"]
Sep 11 15:13:22.996265 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:22.996226    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:13:28.514886 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:28.514849    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:13:37.957035 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:37.956987    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26" podUID="c0a351d5-4cfe-4b87-989b-4d256e2afb71" containerName="main" containerID="cri-o://c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5" gracePeriod=45
Sep 11 15:13:38.210074 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.210009    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:13:38.325671 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.325626    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-model-cache\") pod \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") "
Sep 11 15:13:38.325855 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.325691    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tmp-dir\") pod \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") "
Sep 11 15:13:38.325855 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.325739    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpw4s\" (UniqueName: \"kubernetes.io/projected/c0a351d5-4cfe-4b87-989b-4d256e2afb71-kube-api-access-fpw4s\") pod \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") "
Sep 11 15:13:38.325855 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.325772    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tls-certs\") pod \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") "
Sep 11 15:13:38.325855 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.325794    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-dshm\") pod \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") "
Sep 11 15:13:38.325855 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.325828    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-home\") pod \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\" (UID: \"c0a351d5-4cfe-4b87-989b-4d256e2afb71\") "
Sep 11 15:13:38.326139 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.325968    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0a351d5-4cfe-4b87-989b-4d256e2afb71" (UID: "c0a351d5-4cfe-4b87-989b-4d256e2afb71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:13:38.326139 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.325976    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0a351d5-4cfe-4b87-989b-4d256e2afb71" (UID: "c0a351d5-4cfe-4b87-989b-4d256e2afb71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:13:38.326237 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.326154    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-home" (OuterVolumeSpecName: "home") pod "c0a351d5-4cfe-4b87-989b-4d256e2afb71" (UID: "c0a351d5-4cfe-4b87-989b-4d256e2afb71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:13:38.326237 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.326171    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:38.326237 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.326191    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:38.328056 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.328033    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-dshm" (OuterVolumeSpecName: "dshm") pod "c0a351d5-4cfe-4b87-989b-4d256e2afb71" (UID: "c0a351d5-4cfe-4b87-989b-4d256e2afb71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:13:38.328175 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.328065    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a351d5-4cfe-4b87-989b-4d256e2afb71-kube-api-access-fpw4s" (OuterVolumeSpecName: "kube-api-access-fpw4s") pod "c0a351d5-4cfe-4b87-989b-4d256e2afb71" (UID: "c0a351d5-4cfe-4b87-989b-4d256e2afb71"). InnerVolumeSpecName "kube-api-access-fpw4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:13:38.328175 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.328087    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0a351d5-4cfe-4b87-989b-4d256e2afb71" (UID: "c0a351d5-4cfe-4b87-989b-4d256e2afb71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:13:38.427282 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.427219    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpw4s\" (UniqueName: \"kubernetes.io/projected/c0a351d5-4cfe-4b87-989b-4d256e2afb71-kube-api-access-fpw4s\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:38.427282 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.427280    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a351d5-4cfe-4b87-989b-4d256e2afb71-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:38.427282 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.427294    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:38.427557 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.427302    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a351d5-4cfe-4b87-989b-4d256e2afb71-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:13:38.498898 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.498863    2559 generic.go:356] "Generic (PLEG): container finished" podID="c0a351d5-4cfe-4b87-989b-4d256e2afb71" containerID="c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5" exitCode=0
Sep 11 15:13:38.499075 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.498904    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"
Sep 11 15:13:38.499075 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.498955    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26" event={"ID":"c0a351d5-4cfe-4b87-989b-4d256e2afb71","Type":"ContainerDied","Data":"c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5"}
Sep 11 15:13:38.499075 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.499006    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26" event={"ID":"c0a351d5-4cfe-4b87-989b-4d256e2afb71","Type":"ContainerDied","Data":"445699ea1d5b4f027d28a06a9723e1cb3b9617d170bf79d60dc40e3b5e93f345"}
Sep 11 15:13:38.499075 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.499028    2559 scope.go:117] "RemoveContainer" containerID="c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5"
Sep 11 15:13:38.512160 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.512134    2559 scope.go:117] "RemoveContainer" containerID="c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5"
Sep 11 15:13:38.512902 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:13:38.512876    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5\": container with ID starting with c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5 not found: ID does not exist" containerID="c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5"
Sep 11 15:13:38.513006 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.512913    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5"} err="failed to get container status \"c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5\": rpc error: code = NotFound desc = could not find container \"c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5\": container with ID starting with c49f444e2d06671e6eb998f8c57fda4306cb4583b50fd9da641369f7069ed6b5 not found: ID does not exist"
Sep 11 15:13:38.527672 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.527641    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"]
Sep 11 15:13:38.532830 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:38.532802    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-177c81/ipp-route-177c81-kserve-7489c9bf95-tck26"]
Sep 11 15:13:40.180185 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:40.180138    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0a351d5-4cfe-4b87-989b-4d256e2afb71" path="/var/lib/kubelet/pods/c0a351d5-4cfe-4b87-989b-4d256e2afb71/volumes"
Sep 11 15:13:59.142874 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:13:59.142832    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:00.328635 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:00.328592    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:06.045425 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:06.045383    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:06.145289 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:06.145251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:06.337396 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:06.337303    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:07.079643 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:07.079608    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:07.902396 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:07.902360    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:08.578620 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:08.578584    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:09.513658 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:09.513606    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:10.350948 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:10.350913    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:17.444054 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.443969    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"]
Sep 11 15:14:17.444524 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.444188    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z" podUID="c4f873ce-5342-471a-85b4-2b7ca9bc9455" containerName="maas-api" containerID="cri-o://ee26ab2cb0adadf00a2904cc46eda8c2798d10ed6151b913d6321a1cbcd72880" gracePeriod=30
Sep 11 15:14:17.482364 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.482335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:17.582994 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.582954    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:17.681907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.681872    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:17.691857 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.691825    2559 generic.go:356] "Generic (PLEG): container finished" podID="c4f873ce-5342-471a-85b4-2b7ca9bc9455" containerID="ee26ab2cb0adadf00a2904cc46eda8c2798d10ed6151b913d6321a1cbcd72880" exitCode=0
Sep 11 15:14:17.692020 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.691899    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z" event={"ID":"c4f873ce-5342-471a-85b4-2b7ca9bc9455","Type":"ContainerDied","Data":"ee26ab2cb0adadf00a2904cc46eda8c2798d10ed6151b913d6321a1cbcd72880"}
Sep 11 15:14:17.692020 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.691962    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z" event={"ID":"c4f873ce-5342-471a-85b4-2b7ca9bc9455","Type":"ContainerDied","Data":"9fe35c07e45ff507afc759f2b433129e54f0cea5e99e45ca683e8c5624d5af5f"}
Sep 11 15:14:17.692020 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.691986    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fe35c07e45ff507afc759f2b433129e54f0cea5e99e45ca683e8c5624d5af5f"
Sep 11 15:14:17.703523 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.703441    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:14:17.787567 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.787310    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:17.815355 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.815315    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-metrics-certs\") pod \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") "
Sep 11 15:14:17.815556 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.815389    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2s7k\" (UniqueName: \"kubernetes.io/projected/c4f873ce-5342-471a-85b4-2b7ca9bc9455-kube-api-access-x2s7k\") pod \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") "
Sep 11 15:14:17.815556 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.815484    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f873ce-5342-471a-85b4-2b7ca9bc9455-openshift-service-ca\") pod \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") "
Sep 11 15:14:17.815556 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.815517    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-maas-api-tls\") pod \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\" (UID: \"c4f873ce-5342-471a-85b4-2b7ca9bc9455\") "
Sep 11 15:14:17.815854 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.815825    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f873ce-5342-471a-85b4-2b7ca9bc9455-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4f873ce-5342-471a-85b4-2b7ca9bc9455" (UID: "c4f873ce-5342-471a-85b4-2b7ca9bc9455"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:14:17.817609 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.817572    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4f873ce-5342-471a-85b4-2b7ca9bc9455" (UID: "c4f873ce-5342-471a-85b4-2b7ca9bc9455"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:14:17.817713 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.817635    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f873ce-5342-471a-85b4-2b7ca9bc9455-kube-api-access-x2s7k" (OuterVolumeSpecName: "kube-api-access-x2s7k") pod "c4f873ce-5342-471a-85b4-2b7ca9bc9455" (UID: "c4f873ce-5342-471a-85b4-2b7ca9bc9455"). InnerVolumeSpecName "kube-api-access-x2s7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:14:17.817713 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.817665    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c4f873ce-5342-471a-85b4-2b7ca9bc9455" (UID: "c4f873ce-5342-471a-85b4-2b7ca9bc9455"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:14:17.916230 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.916197    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:14:17.916230 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.916225    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2s7k\" (UniqueName: \"kubernetes.io/projected/c4f873ce-5342-471a-85b4-2b7ca9bc9455-kube-api-access-x2s7k\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:14:17.916230 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.916235    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f873ce-5342-471a-85b4-2b7ca9bc9455-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:14:17.916497 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:17.916244    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f873ce-5342-471a-85b4-2b7ca9bc9455-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:14:18.084155 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:18.084122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:18.183782 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:18.183746    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:18.696987 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:18.696948    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"
Sep 11 15:14:18.721479 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:18.721421    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"]
Sep 11 15:14:18.729413 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:18.729381    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee29a-79f945ffc7-l248z"]
Sep 11 15:14:19.133513 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:19.133443    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:19.904867 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:19.904829    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:20.179704 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:20.179598    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f873ce-5342-471a-85b4-2b7ca9bc9455" path="/var/lib/kubelet/pods/c4f873ce-5342-471a-85b4-2b7ca9bc9455/volumes"
Sep 11 15:14:20.815142 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:20.815103    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:14:23.912841 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:23.912788    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:24.003361 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:24.003322    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:24.197871 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:24.197765    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:24.731825 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:24.731785    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:25.495365 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:25.495328    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:26.309116 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:26.309080    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:27.095166 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:27.095125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:34.999675 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:34.999626    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"]
Sep 11 15:14:35.000194 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:34.999871    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2" podUID="c7c21653-ad82-4871-bca5-505fbf4c70bc" containerName="maas-api" containerID="cri-o://4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223" gracePeriod=30
Sep 11 15:14:35.022206 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.022173    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:35.121435 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.121402    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:35.222251 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.222216    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:35.256118 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.256055    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:14:35.284720 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.284681    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:35.289711 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.289688    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c21653-ad82-4871-bca5-505fbf4c70bc-openshift-service-ca\") pod \"c7c21653-ad82-4871-bca5-505fbf4c70bc\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") "
Sep 11 15:14:35.289835 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.289770    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-maas-api-tls\") pod \"c7c21653-ad82-4871-bca5-505fbf4c70bc\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") "
Sep 11 15:14:35.289835 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.289788    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-metrics-certs\") pod \"c7c21653-ad82-4871-bca5-505fbf4c70bc\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") "
Sep 11 15:14:35.289949 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.289865    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6rdk\" (UniqueName: \"kubernetes.io/projected/c7c21653-ad82-4871-bca5-505fbf4c70bc-kube-api-access-m6rdk\") pod \"c7c21653-ad82-4871-bca5-505fbf4c70bc\" (UID: \"c7c21653-ad82-4871-bca5-505fbf4c70bc\") "
Sep 11 15:14:35.290116 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.290090    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c21653-ad82-4871-bca5-505fbf4c70bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7c21653-ad82-4871-bca5-505fbf4c70bc" (UID: "c7c21653-ad82-4871-bca5-505fbf4c70bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:14:35.292016 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.291979    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c7c21653-ad82-4871-bca5-505fbf4c70bc" (UID: "c7c21653-ad82-4871-bca5-505fbf4c70bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:14:35.292146 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.292026    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7c21653-ad82-4871-bca5-505fbf4c70bc" (UID: "c7c21653-ad82-4871-bca5-505fbf4c70bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:14:35.292146 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.292044    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c21653-ad82-4871-bca5-505fbf4c70bc-kube-api-access-m6rdk" (OuterVolumeSpecName: "kube-api-access-m6rdk") pod "c7c21653-ad82-4871-bca5-505fbf4c70bc" (UID: "c7c21653-ad82-4871-bca5-505fbf4c70bc"). InnerVolumeSpecName "kube-api-access-m6rdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:14:35.391621 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.391583    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:14:35.391621 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.391610    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c21653-ad82-4871-bca5-505fbf4c70bc-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:14:35.391621 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.391620    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6rdk\" (UniqueName: \"kubernetes.io/projected/c7c21653-ad82-4871-bca5-505fbf4c70bc-kube-api-access-m6rdk\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:14:35.391621 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.391630    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c21653-ad82-4871-bca5-505fbf4c70bc-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:14:35.529863 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.529766    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:35.633496 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.633442    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:35.730865 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.730826    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:35.787320 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.787215    2559 generic.go:356] "Generic (PLEG): container finished" podID="c7c21653-ad82-4871-bca5-505fbf4c70bc" containerID="4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223" exitCode=0
Sep 11 15:14:35.787320 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.787270    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"
Sep 11 15:14:35.787320 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.787296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2" event={"ID":"c7c21653-ad82-4871-bca5-505fbf4c70bc","Type":"ContainerDied","Data":"4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223"}
Sep 11 15:14:35.787572 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.787357    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2" event={"ID":"c7c21653-ad82-4871-bca5-505fbf4c70bc","Type":"ContainerDied","Data":"c0ebf0d60d37e98d25cc883c5cd03011702127e75f216e42ec5e73a761db98e2"}
Sep 11 15:14:35.787572 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.787389    2559 scope.go:117] "RemoveContainer" containerID="4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223"
Sep 11 15:14:35.797277 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.797250    2559 scope.go:117] "RemoveContainer" containerID="4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223"
Sep 11 15:14:35.797577 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:14:35.797558    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223\": container with ID starting with 4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223 not found: ID does not exist" containerID="4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223"
Sep 11 15:14:35.797656 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.797591    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223"} err="failed to get container status \"4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223\": rpc error: code = NotFound desc = could not find container \"4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223\": container with ID starting with 4f171438ece649c55cb5487b10c2c51284d249f644fbe40e1333eb4eb3a0a223 not found: ID does not exist"
Sep 11 15:14:35.810051 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.810025    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"]
Sep 11 15:14:35.813961 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:35.813935    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-48898f-59bbdc6c96-snnv2"]
Sep 11 15:14:36.180884 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:36.180799    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7c21653-ad82-4871-bca5-505fbf4c70bc" path="/var/lib/kubelet/pods/c7c21653-ad82-4871-bca5-505fbf4c70bc/volumes"
Sep 11 15:14:36.496318 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:36.496282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:38.219254 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:38.219212    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:14:41.867672 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:41.867625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:41.966563 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:41.966517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:42.196416 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.196313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:42.516326 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516291    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"]
Sep 11 15:14:42.516705 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516691    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f873ce-5342-471a-85b4-2b7ca9bc9455" containerName="maas-api"
Sep 11 15:14:42.516767 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516707    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f873ce-5342-471a-85b4-2b7ca9bc9455" containerName="maas-api"
Sep 11 15:14:42.516767 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516714    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a351d5-4cfe-4b87-989b-4d256e2afb71" containerName="main"
Sep 11 15:14:42.516767 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516720    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a351d5-4cfe-4b87-989b-4d256e2afb71" containerName="main"
Sep 11 15:14:42.516767 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516729    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7c21653-ad82-4871-bca5-505fbf4c70bc" containerName="maas-api"
Sep 11 15:14:42.516767 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516734    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7c21653-ad82-4871-bca5-505fbf4c70bc" containerName="maas-api"
Sep 11 15:14:42.516767 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516755    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy"
Sep 11 15:14:42.516767 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516760    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy"
Sep 11 15:14:42.516767 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516767    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy"
Sep 11 15:14:42.517025 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516772    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy"
Sep 11 15:14:42.517025 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516825    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy"
Sep 11 15:14:42.517025 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516832    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a351d5-4cfe-4b87-989b-4d256e2afb71" containerName="main"
Sep 11 15:14:42.517025 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516842    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f873ce-5342-471a-85b4-2b7ca9bc9455" containerName="maas-api"
Sep 11 15:14:42.517025 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.516847    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7c21653-ad82-4871-bca5-505fbf4c70bc" containerName="maas-api"
Sep 11 15:14:42.521140 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.521108    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"
Sep 11 15:14:42.523688 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.523662    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 15:14:42.523806 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.523738    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\""
Sep 11 15:14:42.527869 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.527836    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"]
Sep 11 15:14:42.565559 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.565518    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de26203f-eb7a-4e0e-b019-ecce24ae8c25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd\" (UID: \"de26203f-eb7a-4e0e-b019-ecce24ae8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"
Sep 11 15:14:42.666798 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.666766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de26203f-eb7a-4e0e-b019-ecce24ae8c25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd\" (UID: \"de26203f-eb7a-4e0e-b019-ecce24ae8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"
Sep 11 15:14:42.667358 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.667337    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de26203f-eb7a-4e0e-b019-ecce24ae8c25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd\" (UID: \"de26203f-eb7a-4e0e-b019-ecce24ae8c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"
Sep 11 15:14:42.690731 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.690690    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:42.834632 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:42.834544    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"
Sep 11 15:14:43.106099 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:43.106071    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"]
Sep 11 15:14:43.108209 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:14:43.108183    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde26203f_eb7a_4e0e_b019_ecce24ae8c25.slice/crio-829519102dcbcceda900b2d489282c6d073220c5cf58902a8f9c8e6985dbe531 WatchSource:0}: Error finding container 829519102dcbcceda900b2d489282c6d073220c5cf58902a8f9c8e6985dbe531: Status 404 returned error can't find the container with id 829519102dcbcceda900b2d489282c6d073220c5cf58902a8f9c8e6985dbe531
Sep 11 15:14:43.573464 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:43.573429    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:43.829344 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:43.829311    2559 generic.go:356] "Generic (PLEG): container finished" podID="de26203f-eb7a-4e0e-b019-ecce24ae8c25" containerID="bde6d588dcef867901e6759d9fa3bb84a6353dc1496031785e54d0aa7f2a2fbc" exitCode=0
Sep 11 15:14:43.829534 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:43.829390    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd" event={"ID":"de26203f-eb7a-4e0e-b019-ecce24ae8c25","Type":"ContainerDied","Data":"bde6d588dcef867901e6759d9fa3bb84a6353dc1496031785e54d0aa7f2a2fbc"}
Sep 11 15:14:43.829534 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:43.829444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd" event={"ID":"de26203f-eb7a-4e0e-b019-ecce24ae8c25","Type":"ContainerStarted","Data":"829519102dcbcceda900b2d489282c6d073220c5cf58902a8f9c8e6985dbe531"}
Sep 11 15:14:44.244172 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:44.244133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:44.968532 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:44.968508    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"
Sep 11 15:14:45.095006 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:45.094901    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de26203f-eb7a-4e0e-b019-ecce24ae8c25-maas-api-service-ca\") pod \"de26203f-eb7a-4e0e-b019-ecce24ae8c25\" (UID: \"de26203f-eb7a-4e0e-b019-ecce24ae8c25\") "
Sep 11 15:14:45.095303 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:45.095277    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de26203f-eb7a-4e0e-b019-ecce24ae8c25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de26203f-eb7a-4e0e-b019-ecce24ae8c25" (UID: "de26203f-eb7a-4e0e-b019-ecce24ae8c25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:14:45.196658 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:45.196614    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de26203f-eb7a-4e0e-b019-ecce24ae8c25-maas-api-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:14:45.840015 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:45.839984    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"
Sep 11 15:14:45.840422 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:45.839977    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd" event={"ID":"de26203f-eb7a-4e0e-b019-ecce24ae8c25","Type":"ContainerDied","Data":"829519102dcbcceda900b2d489282c6d073220c5cf58902a8f9c8e6985dbe531"}
Sep 11 15:14:45.840422 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:45.840068    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="829519102dcbcceda900b2d489282c6d073220c5cf58902a8f9c8e6985dbe531"
Sep 11 15:14:51.732472 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:51.732429    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:52.995910 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:52.995863    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:53.068412 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:53.068363    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:53.130200 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:53.130152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:53.205257 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:53.205219    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"]
Sep 11 15:14:53.208036 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:53.208000    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-928943-6ftpd"]
Sep 11 15:14:53.208208 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:53.208041    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:53.288850 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:53.288740    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:53.592877 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:53.592776    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:53.697576 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:53.697515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:54.180298 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:54.180266    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de26203f-eb7a-4e0e-b019-ecce24ae8c25" path="/var/lib/kubelet/pods/de26203f-eb7a-4e0e-b019-ecce24ae8c25/volumes"
Sep 11 15:14:56.018652 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:56.018619    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:14:59.586579 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:59.586538    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:59.649828 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:59.649787    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:14:59.847350 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:59.847264    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"]
Sep 11 15:14:59.847597 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:59.847570    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy" containerID="cri-o://ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600" gracePeriod=30
Sep 11 15:14:59.925249 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:14:59.925207    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:15:05.103660 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.103628    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:15:05.219944 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.219844    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:15:05.283972 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.283932    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-token\") pod \"4b0414f2-22d7-48f6-996f-43e4bd222e68\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") "
Sep 11 15:15:05.284184 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.283995    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-data\") pod \"4b0414f2-22d7-48f6-996f-43e4bd222e68\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") "
Sep 11 15:15:05.284184 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284019    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-podinfo\") pod \"4b0414f2-22d7-48f6-996f-43e4bd222e68\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") "
Sep 11 15:15:05.284184 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284037    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-certs\") pod \"4b0414f2-22d7-48f6-996f-43e4bd222e68\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") "
Sep 11 15:15:05.284184 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284053    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b0414f2-22d7-48f6-996f-43e4bd222e68-istiod-ca-cert\") pod \"4b0414f2-22d7-48f6-996f-43e4bd222e68\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") "
Sep 11 15:15:05.284184 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284083    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-envoy\") pod \"4b0414f2-22d7-48f6-996f-43e4bd222e68\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") "
Sep 11 15:15:05.284184 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284146    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssh8w\" (UniqueName: \"kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-kube-api-access-ssh8w\") pod \"4b0414f2-22d7-48f6-996f-43e4bd222e68\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") "
Sep 11 15:15:05.284564 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284224    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-socket\") pod \"4b0414f2-22d7-48f6-996f-43e4bd222e68\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") "
Sep 11 15:15:05.284564 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284288    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b0414f2-22d7-48f6-996f-43e4bd222e68" (UID: "4b0414f2-22d7-48f6-996f-43e4bd222e68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:15:05.284564 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284317    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-credential-socket\") pod \"4b0414f2-22d7-48f6-996f-43e4bd222e68\" (UID: \"4b0414f2-22d7-48f6-996f-43e4bd222e68\") "
Sep 11 15:15:05.284564 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284502    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0414f2-22d7-48f6-996f-43e4bd222e68-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b0414f2-22d7-48f6-996f-43e4bd222e68" (UID: "4b0414f2-22d7-48f6-996f-43e4bd222e68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:15:05.284564 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284541    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b0414f2-22d7-48f6-996f-43e4bd222e68" (UID: "4b0414f2-22d7-48f6-996f-43e4bd222e68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:15:05.284815 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284628    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b0414f2-22d7-48f6-996f-43e4bd222e68" (UID: "4b0414f2-22d7-48f6-996f-43e4bd222e68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:15:05.284815 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284676    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b0414f2-22d7-48f6-996f-43e4bd222e68" (UID: "4b0414f2-22d7-48f6-996f-43e4bd222e68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:15:05.284815 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284781    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-data\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:15:05.284815 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284795    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:15:05.284815 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284804    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b0414f2-22d7-48f6-996f-43e4bd222e68-istiod-ca-cert\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:15:05.284815 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284812    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-workload-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:15:05.285129 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.284821    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-credential-socket\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:15:05.286445 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.286419    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b0414f2-22d7-48f6-996f-43e4bd222e68" (UID: "4b0414f2-22d7-48f6-996f-43e4bd222e68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:15:05.286592 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.286486    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b0414f2-22d7-48f6-996f-43e4bd222e68" (UID: "4b0414f2-22d7-48f6-996f-43e4bd222e68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 15:15:05.286772 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.286743    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-kube-api-access-ssh8w" (OuterVolumeSpecName: "kube-api-access-ssh8w") pod "4b0414f2-22d7-48f6-996f-43e4bd222e68" (UID: "4b0414f2-22d7-48f6-996f-43e4bd222e68"). InnerVolumeSpecName "kube-api-access-ssh8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:15:05.286869 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.286814    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b0414f2-22d7-48f6-996f-43e4bd222e68" (UID: "4b0414f2-22d7-48f6-996f-43e4bd222e68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:15:05.385963 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.385916    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssh8w\" (UniqueName: \"kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-kube-api-access-ssh8w\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:15:05.385963 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.385953    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-token\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:15:05.385963 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.385963    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-podinfo\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:15:05.385963 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.385972    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b0414f2-22d7-48f6-996f-43e4bd222e68-istio-envoy\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:15:05.936021 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.935983    2559 generic.go:356] "Generic (PLEG): container finished" podID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerID="ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600" exitCode=0
Sep 11 15:15:05.936197 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.936039    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"
Sep 11 15:15:05.936197 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.936068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" event={"ID":"4b0414f2-22d7-48f6-996f-43e4bd222e68","Type":"ContainerDied","Data":"ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600"}
Sep 11 15:15:05.936197 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.936124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78" event={"ID":"4b0414f2-22d7-48f6-996f-43e4bd222e68","Type":"ContainerDied","Data":"b01d24db2869205ca98037eb85fc2e27d547490da8bdae88a455a497501c04e0"}
Sep 11 15:15:05.936197 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.936146    2559 scope.go:117] "RemoveContainer" containerID="ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600"
Sep 11 15:15:05.945669 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.945649    2559 scope.go:117] "RemoveContainer" containerID="9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948"
Sep 11 15:15:05.954073 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.953988    2559 scope.go:117] "RemoveContainer" containerID="ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600"
Sep 11 15:15:05.954326 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:15:05.954303    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600\": container with ID starting with ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600 not found: ID does not exist" containerID="ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600"
Sep 11 15:15:05.954384 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.954335    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600"} err="failed to get container status \"ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600\": rpc error: code = NotFound desc = could not find container \"ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600\": container with ID starting with ed5dfe330f1cb35117bc01ac16f55aec565a2fe90538ad0e8ccaecc18da0f600 not found: ID does not exist"
Sep 11 15:15:05.954384 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.954352    2559 scope.go:117] "RemoveContainer" containerID="9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948"
Sep 11 15:15:05.954628 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:15:05.954610    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948\": container with ID starting with 9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948 not found: ID does not exist" containerID="9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948"
Sep 11 15:15:05.954675 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.954632    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948"} err="failed to get container status \"9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948\": rpc error: code = NotFound desc = could not find container \"9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948\": container with ID starting with 9faf3a951cd0dae23465a1ffeb2bba7207fb3927ae5a062ad582163f07988948 not found: ID does not exist"
Sep 11 15:15:05.958953 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.958930    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"]
Sep 11 15:15:05.964086 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:05.964062    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-928943-openshift-default-58b4db468d-6xh78"]
Sep 11 15:15:06.180938 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:06.180903    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" path="/var/lib/kubelet/pods/4b0414f2-22d7-48f6-996f-43e4bd222e68/volumes"
Sep 11 15:15:15.028079 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:15.028044    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:15:21.658946 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:15:21.658905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:16:08.051672 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:16:08.051637    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:16:09.489188 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:16:09.489148    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:16:18.878526 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:16:18.878479    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:16:18.979096 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:16:18.979050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:16:25.720020 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:16:25.719982    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:16:26.220234 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:16:26.220203    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:16:27.776894 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:16:27.776853    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:16:54.114282 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:16:54.114235    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:16:54.577142 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:16:54.577111    2559 scope.go:117] "RemoveContainer" containerID="ee26ab2cb0adadf00a2904cc46eda8c2798d10ed6151b913d6321a1cbcd72880"
Sep 11 15:16:56.712593 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:16:56.712550    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:16:56.768479 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:16:56.768425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:17:01.823751 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:01.823705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:17:09.210120 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:09.210079    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:17:16.610391 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:16.610309    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:17:31.913352 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:31.913314    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:17:39.711481 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:39.711437    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:17:45.332240 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:45.332201    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"]
Sep 11 15:17:45.332702 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:45.332411    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n" podUID="e61d46f5-9367-4e10-90bb-3aadda2a5486" containerName="manager" containerID="cri-o://ed92f160316592d1ad173168b9bd6130c6c345c6f77f18ce0f20ed7d4913f6bd" gracePeriod=10
Sep 11 15:17:45.691524 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:45.691488    2559 generic.go:356] "Generic (PLEG): container finished" podID="e61d46f5-9367-4e10-90bb-3aadda2a5486" containerID="ed92f160316592d1ad173168b9bd6130c6c345c6f77f18ce0f20ed7d4913f6bd" exitCode=0
Sep 11 15:17:45.691700 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:45.691544    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n" event={"ID":"e61d46f5-9367-4e10-90bb-3aadda2a5486","Type":"ContainerDied","Data":"ed92f160316592d1ad173168b9bd6130c6c345c6f77f18ce0f20ed7d4913f6bd"}
Sep 11 15:17:46.298724 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.298688    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 15:17:46.366902 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.366864    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e61d46f5-9367-4e10-90bb-3aadda2a5486-extensions-socket-volume\") pod \"e61d46f5-9367-4e10-90bb-3aadda2a5486\" (UID: \"e61d46f5-9367-4e10-90bb-3aadda2a5486\") "
Sep 11 15:17:46.367349 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.366943    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bff8j\" (UniqueName: \"kubernetes.io/projected/e61d46f5-9367-4e10-90bb-3aadda2a5486-kube-api-access-bff8j\") pod \"e61d46f5-9367-4e10-90bb-3aadda2a5486\" (UID: \"e61d46f5-9367-4e10-90bb-3aadda2a5486\") "
Sep 11 15:17:46.367409 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.367381    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61d46f5-9367-4e10-90bb-3aadda2a5486-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e61d46f5-9367-4e10-90bb-3aadda2a5486" (UID: "e61d46f5-9367-4e10-90bb-3aadda2a5486"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:17:46.369227 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.369191    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61d46f5-9367-4e10-90bb-3aadda2a5486-kube-api-access-bff8j" (OuterVolumeSpecName: "kube-api-access-bff8j") pod "e61d46f5-9367-4e10-90bb-3aadda2a5486" (UID: "e61d46f5-9367-4e10-90bb-3aadda2a5486"). InnerVolumeSpecName "kube-api-access-bff8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:17:46.468738 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.468642    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bff8j\" (UniqueName: \"kubernetes.io/projected/e61d46f5-9367-4e10-90bb-3aadda2a5486-kube-api-access-bff8j\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:17:46.468738 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.468676    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e61d46f5-9367-4e10-90bb-3aadda2a5486-extensions-socket-volume\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:17:46.697773 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.697727    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n" event={"ID":"e61d46f5-9367-4e10-90bb-3aadda2a5486","Type":"ContainerDied","Data":"df924e051c740c1b663cb2acf0983f5e238a0424eb1d990fae548c8ec6cb6ff7"}
Sep 11 15:17:46.697968 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.697748    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"
Sep 11 15:17:46.697968 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.697816    2559 scope.go:117] "RemoveContainer" containerID="ed92f160316592d1ad173168b9bd6130c6c345c6f77f18ce0f20ed7d4913f6bd"
Sep 11 15:17:46.719956 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.719887    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"]
Sep 11 15:17:46.723321 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:46.723287    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r6m6n"]
Sep 11 15:17:48.179417 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:17:48.179374    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e61d46f5-9367-4e10-90bb-3aadda2a5486" path="/var/lib/kubelet/pods/e61d46f5-9367-4e10-90bb-3aadda2a5486/volumes"
Sep 11 15:18:27.626163 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:18:27.626129    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:19:08.063776 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:08.063745    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:19:14.670573 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:14.670535    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:19:24.082516 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:24.082481    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:19:38.071287 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071253    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"]
Sep 11 15:19:38.071737 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071663    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy"
Sep 11 15:19:38.071737 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071678    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy"
Sep 11 15:19:38.071737 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071689    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy"
Sep 11 15:19:38.071737 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071695    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy"
Sep 11 15:19:38.071737 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071707    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61d46f5-9367-4e10-90bb-3aadda2a5486" containerName="manager"
Sep 11 15:19:38.071737 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071713    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61d46f5-9367-4e10-90bb-3aadda2a5486" containerName="manager"
Sep 11 15:19:38.071737 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071725    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de26203f-eb7a-4e0e-b019-ecce24ae8c25" containerName="revoke-keys"
Sep 11 15:19:38.071737 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071733    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="de26203f-eb7a-4e0e-b019-ecce24ae8c25" containerName="revoke-keys"
Sep 11 15:19:38.071996 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071801    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="16bc768f-a604-4d60-a7e6-17783dd28cbc" containerName="istio-proxy"
Sep 11 15:19:38.071996 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071812    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="de26203f-eb7a-4e0e-b019-ecce24ae8c25" containerName="revoke-keys"
Sep 11 15:19:38.071996 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071819    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy"
Sep 11 15:19:38.071996 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071826    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61d46f5-9367-4e10-90bb-3aadda2a5486" containerName="manager"
Sep 11 15:19:38.071996 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.071834    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0414f2-22d7-48f6-996f-43e4bd222e68" containerName="istio-proxy"
Sep 11 15:19:38.075013 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.074994    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.077672 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.077644    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c41cd557-kserve-self-signed-certs\""
Sep 11 15:19:38.083125 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.083102    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"]
Sep 11 15:19:38.095368 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.095343    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:19:38.196040 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.196005    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tmp-dir\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.196201 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.196051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tls-certs\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.196201 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.196094    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-home\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.196201 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.196127    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62lww\" (UniqueName: \"kubernetes.io/projected/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-kube-api-access-62lww\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.196201 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.196153    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-model-cache\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.196201 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.196182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-dshm\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.297407 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.297368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tmp-dir\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.297407 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.297413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tls-certs\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.297699 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.297433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-home\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.297699 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.297492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lww\" (UniqueName: \"kubernetes.io/projected/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-kube-api-access-62lww\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.297699 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.297514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-model-cache\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.297699 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.297540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-dshm\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.297931 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.297856    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tmp-dir\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.297931 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.297884    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-home\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.298045 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.297996    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-model-cache\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.299778 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.299755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-dshm\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.299955 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.299939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tls-certs\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.305836 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.305812    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62lww\" (UniqueName: \"kubernetes.io/projected/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-kube-api-access-62lww\") pod \"e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") " pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.384130 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.384040    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:38.656326 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:38.656294    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"]
Sep 11 15:19:38.658448 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:19:38.658414    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ae33a17_2b72_46e8_ba2b_3e0321d4295d.slice/crio-04c3d30defdd3e0d735af9aaf4d88c5f72a46d8e2d9f8b0ec42cfa15d3e90434 WatchSource:0}: Error finding container 04c3d30defdd3e0d735af9aaf4d88c5f72a46d8e2d9f8b0ec42cfa15d3e90434: Status 404 returned error can't find the container with id 04c3d30defdd3e0d735af9aaf4d88c5f72a46d8e2d9f8b0ec42cfa15d3e90434
Sep 11 15:19:39.214003 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:39.213968    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg" event={"ID":"6ae33a17-2b72-46e8-ba2b-3e0321d4295d","Type":"ContainerStarted","Data":"c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a"}
Sep 11 15:19:39.214003 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:39.214009    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg" event={"ID":"6ae33a17-2b72-46e8-ba2b-3e0321d4295d","Type":"ContainerStarted","Data":"04c3d30defdd3e0d735af9aaf4d88c5f72a46d8e2d9f8b0ec42cfa15d3e90434"}
Sep 11 15:19:39.234516 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:39.234446    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg" podStartSLOduration=1.234428645 podStartE2EDuration="1.234428645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:19:39.232175226 +0000 UTC m=+2385.554803598" watchObservedRunningTime="2026-09-11 15:19:39.234428645 +0000 UTC m=+2385.557057013"
Sep 11 15:19:40.672783 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:40.672747    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:19:45.463474 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.463421    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"]
Sep 11 15:19:45.466908 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.466881    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.469564 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.469538    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c41cd557-kserve-self-signed-certs\""
Sep 11 15:19:45.476639 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.476614    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"]
Sep 11 15:19:45.499237 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.499205    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:19:45.568519 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.568448    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tmp-dir\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.568700 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.568534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tls-certs\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.568700 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.568556    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwkd\" (UniqueName: \"kubernetes.io/projected/8988c06e-6faf-4514-b8f8-65571ffaeb6b-kube-api-access-qqwkd\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.568700 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.568625    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-model-cache\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.568700 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.568646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-home\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.568834 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.568726    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-dshm\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.669825 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.669769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-home\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.670027 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.669864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-dshm\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.670027 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.669906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tmp-dir\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.670027 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.669927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tls-certs\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.670027 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.669946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwkd\" (UniqueName: \"kubernetes.io/projected/8988c06e-6faf-4514-b8f8-65571ffaeb6b-kube-api-access-qqwkd\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.670027 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.670015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-model-cache\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.670304 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.670220    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-home\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.670304 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.670248    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tmp-dir\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.670396 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.670354    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-model-cache\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.672193 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.672175    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-dshm\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.672512 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.672490    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tls-certs\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.678667 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.678638    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwkd\" (UniqueName: \"kubernetes.io/projected/8988c06e-6faf-4514-b8f8-65571ffaeb6b-kube-api-access-qqwkd\") pod \"e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") " pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:45.778745 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:45.778705    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:46.050988 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:46.050962    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"]
Sep 11 15:19:46.053533 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:19:46.053505    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8988c06e_6faf_4514_b8f8_65571ffaeb6b.slice/crio-6110d9e4dcdc9fde0760cd2e9afa4066208933980bc310656f90bf393959d079 WatchSource:0}: Error finding container 6110d9e4dcdc9fde0760cd2e9afa4066208933980bc310656f90bf393959d079: Status 404 returned error can't find the container with id 6110d9e4dcdc9fde0760cd2e9afa4066208933980bc310656f90bf393959d079
Sep 11 15:19:46.245396 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:46.245361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw" event={"ID":"8988c06e-6faf-4514-b8f8-65571ffaeb6b","Type":"ContainerStarted","Data":"5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69"}
Sep 11 15:19:46.245623 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:46.245409    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw" event={"ID":"8988c06e-6faf-4514-b8f8-65571ffaeb6b","Type":"ContainerStarted","Data":"6110d9e4dcdc9fde0760cd2e9afa4066208933980bc310656f90bf393959d079"}
Sep 11 15:19:46.263636 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:46.263585    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw" podStartSLOduration=1.263568733 podStartE2EDuration="1.263568733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:19:46.261608652 +0000 UTC m=+2392.584237021" watchObservedRunningTime="2026-09-11 15:19:46.263568733 +0000 UTC m=+2392.586197101"
Sep 11 15:19:48.276311 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:48.276272    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:19:48.385084 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:48.385047    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:48.385254 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:48.385209    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:48.399474 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:48.399422    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:49.271355 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:49.271321    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:19:55.779499 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:55.779433    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:55.779899 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:55.779521    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:55.791863 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:55.791833    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:19:56.304450 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:19:56.304417    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:20:22.563113 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:22.563033    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"]
Sep 11 15:20:22.581264 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:22.581226    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:20:23.040860 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:23.040826    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"]
Sep 11 15:20:23.061266 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:23.061231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:20:25.474516 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:25.474480    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:20:33.028847 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:33.028813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:20:34.468748 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:34.468710    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:20:37.582816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.582754    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg" podUID="6ae33a17-2b72-46e8-ba2b-3e0321d4295d" containerName="main" containerID="cri-o://c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a" gracePeriod=45
Sep 11 15:20:37.832006 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.831977    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:20:37.951051 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.950965    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tmp-dir\") pod \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") "
Sep 11 15:20:37.951051 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.951007    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-model-cache\") pod \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") "
Sep 11 15:20:37.951051 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.951032    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-dshm\") pod \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") "
Sep 11 15:20:37.951325 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.951082    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-home\") pod \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") "
Sep 11 15:20:37.951325 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.951141    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62lww\" (UniqueName: \"kubernetes.io/projected/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-kube-api-access-62lww\") pod \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") "
Sep 11 15:20:37.951325 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.951170    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tls-certs\") pod \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\" (UID: \"6ae33a17-2b72-46e8-ba2b-3e0321d4295d\") "
Sep 11 15:20:37.951325 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.951277    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ae33a17-2b72-46e8-ba2b-3e0321d4295d" (UID: "6ae33a17-2b72-46e8-ba2b-3e0321d4295d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:20:37.951325 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.951290    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ae33a17-2b72-46e8-ba2b-3e0321d4295d" (UID: "6ae33a17-2b72-46e8-ba2b-3e0321d4295d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:20:37.951625 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.951470    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-home" (OuterVolumeSpecName: "home") pod "6ae33a17-2b72-46e8-ba2b-3e0321d4295d" (UID: "6ae33a17-2b72-46e8-ba2b-3e0321d4295d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:20:37.951625 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.951563    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:37.951625 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.951580    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:37.951625 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.951588    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:37.953268 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.953240    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-dshm" (OuterVolumeSpecName: "dshm") pod "6ae33a17-2b72-46e8-ba2b-3e0321d4295d" (UID: "6ae33a17-2b72-46e8-ba2b-3e0321d4295d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:20:37.953377 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.953280    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ae33a17-2b72-46e8-ba2b-3e0321d4295d" (UID: "6ae33a17-2b72-46e8-ba2b-3e0321d4295d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:20:37.953377 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:37.953315    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-kube-api-access-62lww" (OuterVolumeSpecName: "kube-api-access-62lww") pod "6ae33a17-2b72-46e8-ba2b-3e0321d4295d" (UID: "6ae33a17-2b72-46e8-ba2b-3e0321d4295d"). InnerVolumeSpecName "kube-api-access-62lww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:20:38.053105 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.053069    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62lww\" (UniqueName: \"kubernetes.io/projected/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-kube-api-access-62lww\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:38.053105 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.053100    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:38.053105 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.053110    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae33a17-2b72-46e8-ba2b-3e0321d4295d-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:38.059822 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.059779    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw" podUID="8988c06e-6faf-4514-b8f8-65571ffaeb6b" containerName="main" containerID="cri-o://5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69" gracePeriod=45
Sep 11 15:20:38.307805 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.307775    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:20:38.356372 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356338    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-model-cache\") pod \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") "
Sep 11 15:20:38.356561 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356395    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tmp-dir\") pod \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") "
Sep 11 15:20:38.356561 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356445    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-home\") pod \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") "
Sep 11 15:20:38.356561 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356527    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqwkd\" (UniqueName: \"kubernetes.io/projected/8988c06e-6faf-4514-b8f8-65571ffaeb6b-kube-api-access-qqwkd\") pod \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") "
Sep 11 15:20:38.356561 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356551    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-dshm\") pod \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") "
Sep 11 15:20:38.356792 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356593    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tls-certs\") pod \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\" (UID: \"8988c06e-6faf-4514-b8f8-65571ffaeb6b\") "
Sep 11 15:20:38.356792 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356721    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8988c06e-6faf-4514-b8f8-65571ffaeb6b" (UID: "8988c06e-6faf-4514-b8f8-65571ffaeb6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:20:38.356792 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356728    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8988c06e-6faf-4514-b8f8-65571ffaeb6b" (UID: "8988c06e-6faf-4514-b8f8-65571ffaeb6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:20:38.356792 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356761    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-home" (OuterVolumeSpecName: "home") pod "8988c06e-6faf-4514-b8f8-65571ffaeb6b" (UID: "8988c06e-6faf-4514-b8f8-65571ffaeb6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:20:38.357011 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356900    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-model-cache\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:38.357011 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356918    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tmp-dir\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:38.357011 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.356928    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-home\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:38.358736 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.358703    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-dshm" (OuterVolumeSpecName: "dshm") pod "8988c06e-6faf-4514-b8f8-65571ffaeb6b" (UID: "8988c06e-6faf-4514-b8f8-65571ffaeb6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:20:38.358736 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.358706    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8988c06e-6faf-4514-b8f8-65571ffaeb6b" (UID: "8988c06e-6faf-4514-b8f8-65571ffaeb6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:20:38.358889 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.358762    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8988c06e-6faf-4514-b8f8-65571ffaeb6b-kube-api-access-qqwkd" (OuterVolumeSpecName: "kube-api-access-qqwkd") pod "8988c06e-6faf-4514-b8f8-65571ffaeb6b" (UID: "8988c06e-6faf-4514-b8f8-65571ffaeb6b"). InnerVolumeSpecName "kube-api-access-qqwkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:20:38.458383 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.458348    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqwkd\" (UniqueName: \"kubernetes.io/projected/8988c06e-6faf-4514-b8f8-65571ffaeb6b-kube-api-access-qqwkd\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:38.458383 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.458376    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8988c06e-6faf-4514-b8f8-65571ffaeb6b-dshm\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:38.458383 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.458386    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8988c06e-6faf-4514-b8f8-65571ffaeb6b-tls-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:38.489422 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.489335    2559 generic.go:356] "Generic (PLEG): container finished" podID="6ae33a17-2b72-46e8-ba2b-3e0321d4295d" containerID="c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a" exitCode=0
Sep 11 15:20:38.489422 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.489377    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"
Sep 11 15:20:38.489647 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.489420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg" event={"ID":"6ae33a17-2b72-46e8-ba2b-3e0321d4295d","Type":"ContainerDied","Data":"c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a"}
Sep 11 15:20:38.489647 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.489488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg" event={"ID":"6ae33a17-2b72-46e8-ba2b-3e0321d4295d","Type":"ContainerDied","Data":"04c3d30defdd3e0d735af9aaf4d88c5f72a46d8e2d9f8b0ec42cfa15d3e90434"}
Sep 11 15:20:38.489647 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.489506    2559 scope.go:117] "RemoveContainer" containerID="c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a"
Sep 11 15:20:38.490805 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.490784    2559 generic.go:356] "Generic (PLEG): container finished" podID="8988c06e-6faf-4514-b8f8-65571ffaeb6b" containerID="5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69" exitCode=0
Sep 11 15:20:38.490925 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.490827    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"
Sep 11 15:20:38.490925 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.490849    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw" event={"ID":"8988c06e-6faf-4514-b8f8-65571ffaeb6b","Type":"ContainerDied","Data":"5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69"}
Sep 11 15:20:38.490925 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.490877    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw" event={"ID":"8988c06e-6faf-4514-b8f8-65571ffaeb6b","Type":"ContainerDied","Data":"6110d9e4dcdc9fde0760cd2e9afa4066208933980bc310656f90bf393959d079"}
Sep 11 15:20:38.503849 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.503790    2559 scope.go:117] "RemoveContainer" containerID="c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a"
Sep 11 15:20:38.504176 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:20:38.504147    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a\": container with ID starting with c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a not found: ID does not exist" containerID="c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a"
Sep 11 15:20:38.504285 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.504187    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a"} err="failed to get container status \"c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a\": rpc error: code = NotFound desc = could not find container \"c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a\": container with ID starting with c0dc3ab7a0f5fff79d6939e2fe4f7b99b5c38da5fcd03c88d1d34bee9181709a not found: ID does not exist"
Sep 11 15:20:38.504285 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.504212    2559 scope.go:117] "RemoveContainer" containerID="5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69"
Sep 11 15:20:38.512373 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.512347    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"]
Sep 11 15:20:38.512640 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.512618    2559 scope.go:117] "RemoveContainer" containerID="5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69"
Sep 11 15:20:38.512952 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:20:38.512935    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69\": container with ID starting with 5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69 not found: ID does not exist" containerID="5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69"
Sep 11 15:20:38.513007 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.512958    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69"} err="failed to get container status \"5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69\": rpc error: code = NotFound desc = could not find container \"5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69\": container with ID starting with 5312e1392c1fe599aedc58375dd62f45f205f5c2865c96e97ccabecad98fdd69 not found: ID does not exist"
Sep 11 15:20:38.517279 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.517255    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c41cd557-kserve-5f94fb9f95-mgbcg"]
Sep 11 15:20:38.530131 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.530103    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"]
Sep 11 15:20:38.533619 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:38.533594    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c41cd557-kserve-ccdf749f5-tbgbw"]
Sep 11 15:20:39.769615 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:39.769575    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:20:40.180068 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.179983    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ae33a17-2b72-46e8-ba2b-3e0321d4295d" path="/var/lib/kubelet/pods/6ae33a17-2b72-46e8-ba2b-3e0321d4295d/volumes"
Sep 11 15:20:40.180378 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.180365    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8988c06e-6faf-4514-b8f8-65571ffaeb6b" path="/var/lib/kubelet/pods/8988c06e-6faf-4514-b8f8-65571ffaeb6b/volumes"
Sep 11 15:20:40.278813 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.278774    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"]
Sep 11 15:20:40.279135 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.279123    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae33a17-2b72-46e8-ba2b-3e0321d4295d" containerName="main"
Sep 11 15:20:40.279177 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.279137    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae33a17-2b72-46e8-ba2b-3e0321d4295d" containerName="main"
Sep 11 15:20:40.279177 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.279148    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8988c06e-6faf-4514-b8f8-65571ffaeb6b" containerName="main"
Sep 11 15:20:40.279177 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.279154    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8988c06e-6faf-4514-b8f8-65571ffaeb6b" containerName="main"
Sep 11 15:20:40.279275 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.279219    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8988c06e-6faf-4514-b8f8-65571ffaeb6b" containerName="main"
Sep 11 15:20:40.279275 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.279228    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ae33a17-2b72-46e8-ba2b-3e0321d4295d" containerName="main"
Sep 11 15:20:40.283648 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.283626    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.287243 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.287206    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 15:20:40.287386 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.287283    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 15:20:40.287386 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.287308    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzqqg\""
Sep 11 15:20:40.287386 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.287342    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 15:20:40.287386 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.287225    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 15:20:40.292767 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.292741    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"]
Sep 11 15:20:40.300222 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.300199    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"]
Sep 11 15:20:40.303587 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.303569    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.313225 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.313197    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"]
Sep 11 15:20:40.374131 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.374099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-openshift-service-ca\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.374131 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.374134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vplbr\" (UniqueName: \"kubernetes.io/projected/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-kube-api-access-vplbr\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.374401 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.374164    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cca1259-bec7-4714-95fe-7157e3767fa8-openshift-service-ca\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.374401 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.374247    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klsvr\" (UniqueName: \"kubernetes.io/projected/7cca1259-bec7-4714-95fe-7157e3767fa8-kube-api-access-klsvr\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.374401 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.374307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-maas-api-tls\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.374401 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.374353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-metrics-certs\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.374628 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.374411    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-metrics-certs\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.374628 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.374512    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-maas-api-tls\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.475502 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.475472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klsvr\" (UniqueName: \"kubernetes.io/projected/7cca1259-bec7-4714-95fe-7157e3767fa8-kube-api-access-klsvr\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.475693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.475512    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-maas-api-tls\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.475693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.475548    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-metrics-certs\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.475693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.475595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-metrics-certs\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.475693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.475640    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-maas-api-tls\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.475693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.475664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-openshift-service-ca\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.475693 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.475684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vplbr\" (UniqueName: \"kubernetes.io/projected/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-kube-api-access-vplbr\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.476033 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.475719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cca1259-bec7-4714-95fe-7157e3767fa8-openshift-service-ca\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.476476 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.476437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-openshift-service-ca\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.476591 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.476529    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cca1259-bec7-4714-95fe-7157e3767fa8-openshift-service-ca\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.478178 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.478152    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-maas-api-tls\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.478259 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.478183    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-metrics-certs\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.478391 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.478374    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-metrics-certs\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.478495 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.478474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-maas-api-tls\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.483332 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.483308    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vplbr\" (UniqueName: \"kubernetes.io/projected/06536d2f-4a2f-4eca-a2e7-f88bf17cc92c-kube-api-access-vplbr\") pod \"maas-api-7879c6fd8-czqxz\" (UID: \"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.483418 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.483311    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klsvr\" (UniqueName: \"kubernetes.io/projected/7cca1259-bec7-4714-95fe-7157e3767fa8-kube-api-access-klsvr\") pod \"maas-api-56956d4668-r6wbg\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.595704 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.595662    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:40.616045 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.616010    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:40.867893 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.867866    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"]
Sep 11 15:20:40.869979 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:20:40.869951    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cca1259_bec7_4714_95fe_7157e3767fa8.slice/crio-d1ac42d3d8909a0946acb69e09fcaafde02349e6fb36009a579fb35243e276d3 WatchSource:0}: Error finding container d1ac42d3d8909a0946acb69e09fcaafde02349e6fb36009a579fb35243e276d3: Status 404 returned error can't find the container with id d1ac42d3d8909a0946acb69e09fcaafde02349e6fb36009a579fb35243e276d3
Sep 11 15:20:40.871268 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.871247    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 15:20:40.895860 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:40.895836    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"]
Sep 11 15:20:40.897879 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:20:40.897854    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06536d2f_4a2f_4eca_a2e7_f88bf17cc92c.slice/crio-fcf091099a59acf9aa1dbafbcb1c0ad013df56a62cb3e249090e675eaab9ee40 WatchSource:0}: Error finding container fcf091099a59acf9aa1dbafbcb1c0ad013df56a62cb3e249090e675eaab9ee40: Status 404 returned error can't find the container with id fcf091099a59acf9aa1dbafbcb1c0ad013df56a62cb3e249090e675eaab9ee40
Sep 11 15:20:41.510831 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:41.510789    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz" event={"ID":"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c","Type":"ContainerStarted","Data":"4698d710fc3ced7125b71388993019df35ae0a96986dc953ea9891c58786b1fd"}
Sep 11 15:20:41.511027 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:41.510853    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz" event={"ID":"06536d2f-4a2f-4eca-a2e7-f88bf17cc92c","Type":"ContainerStarted","Data":"fcf091099a59acf9aa1dbafbcb1c0ad013df56a62cb3e249090e675eaab9ee40"}
Sep 11 15:20:41.511027 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:41.510882    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:41.512475 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:41.512422    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg" event={"ID":"7cca1259-bec7-4714-95fe-7157e3767fa8","Type":"ContainerStarted","Data":"48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62"}
Sep 11 15:20:41.512572 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:41.512516    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:41.512572 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:41.512544    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg" event={"ID":"7cca1259-bec7-4714-95fe-7157e3767fa8","Type":"ContainerStarted","Data":"d1ac42d3d8909a0946acb69e09fcaafde02349e6fb36009a579fb35243e276d3"}
Sep 11 15:20:41.528679 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:41.528207    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz" podStartSLOduration=1.098308915 podStartE2EDuration="1.528191433s" totalImagesPullingTime="429.882518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:20:41.527639272 +0000 UTC m=+2447.850267642" watchObservedRunningTime="2026-09-11 15:20:41.528191433 +0000 UTC m=+2447.850819801"
Sep 11 15:20:41.545907 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:41.545855    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg" podStartSLOduration=0.999939445 podStartE2EDuration="1.545840339s" totalImagesPullingTime="545.900894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:20:41.54417048 +0000 UTC m=+2447.866798852" watchObservedRunningTime="2026-09-11 15:20:41.545840339 +0000 UTC m=+2447.868468707"
Sep 11 15:20:47.523276 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.523243    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-czqxz"
Sep 11 15:20:47.523747 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.523717    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:47.587472 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.587425    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"]
Sep 11 15:20:47.587697 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.587666    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg" podUID="7cca1259-bec7-4714-95fe-7157e3767fa8" containerName="maas-api" containerID="cri-o://48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62" gracePeriod=30
Sep 11 15:20:47.831839 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.831810    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:47.946686 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.946644    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-metrics-certs\") pod \"7cca1259-bec7-4714-95fe-7157e3767fa8\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") "
Sep 11 15:20:47.946873 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.946758    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-maas-api-tls\") pod \"7cca1259-bec7-4714-95fe-7157e3767fa8\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") "
Sep 11 15:20:47.946873 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.946814    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cca1259-bec7-4714-95fe-7157e3767fa8-openshift-service-ca\") pod \"7cca1259-bec7-4714-95fe-7157e3767fa8\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") "
Sep 11 15:20:47.946873 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.946841    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klsvr\" (UniqueName: \"kubernetes.io/projected/7cca1259-bec7-4714-95fe-7157e3767fa8-kube-api-access-klsvr\") pod \"7cca1259-bec7-4714-95fe-7157e3767fa8\" (UID: \"7cca1259-bec7-4714-95fe-7157e3767fa8\") "
Sep 11 15:20:47.947203 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.947168    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cca1259-bec7-4714-95fe-7157e3767fa8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7cca1259-bec7-4714-95fe-7157e3767fa8" (UID: "7cca1259-bec7-4714-95fe-7157e3767fa8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:20:47.948932 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.948903    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7cca1259-bec7-4714-95fe-7157e3767fa8" (UID: "7cca1259-bec7-4714-95fe-7157e3767fa8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:20:47.949026 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.948978    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cca1259-bec7-4714-95fe-7157e3767fa8-kube-api-access-klsvr" (OuterVolumeSpecName: "kube-api-access-klsvr") pod "7cca1259-bec7-4714-95fe-7157e3767fa8" (UID: "7cca1259-bec7-4714-95fe-7157e3767fa8"). InnerVolumeSpecName "kube-api-access-klsvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:20:47.949026 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:47.949010    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7cca1259-bec7-4714-95fe-7157e3767fa8" (UID: "7cca1259-bec7-4714-95fe-7157e3767fa8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:20:48.047875 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.047836    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-maas-api-tls\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:48.047875 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.047867    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cca1259-bec7-4714-95fe-7157e3767fa8-openshift-service-ca\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:48.047875 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.047876    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klsvr\" (UniqueName: \"kubernetes.io/projected/7cca1259-bec7-4714-95fe-7157e3767fa8-kube-api-access-klsvr\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:48.047875 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.047885    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cca1259-bec7-4714-95fe-7157e3767fa8-metrics-certs\") on node \"ip-10-0-128-223.ec2.internal\" DevicePath \"\""
Sep 11 15:20:48.270344 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.270304    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:20:48.548637 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.548602    2559 generic.go:356] "Generic (PLEG): container finished" podID="7cca1259-bec7-4714-95fe-7157e3767fa8" containerID="48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62" exitCode=0
Sep 11 15:20:48.549109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.548647    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"
Sep 11 15:20:48.549109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.548682    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg" event={"ID":"7cca1259-bec7-4714-95fe-7157e3767fa8","Type":"ContainerDied","Data":"48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62"}
Sep 11 15:20:48.549109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.548734    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56956d4668-r6wbg" event={"ID":"7cca1259-bec7-4714-95fe-7157e3767fa8","Type":"ContainerDied","Data":"d1ac42d3d8909a0946acb69e09fcaafde02349e6fb36009a579fb35243e276d3"}
Sep 11 15:20:48.549109 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.548755    2559 scope.go:117] "RemoveContainer" containerID="48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62"
Sep 11 15:20:48.557393 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.557376    2559 scope.go:117] "RemoveContainer" containerID="48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62"
Sep 11 15:20:48.557733 ip-10-0-128-223 kubenswrapper[2559]: E0911 15:20:48.557712    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62\": container with ID starting with 48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62 not found: ID does not exist" containerID="48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62"
Sep 11 15:20:48.557788 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.557741    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62"} err="failed to get container status \"48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62\": rpc error: code = NotFound desc = could not find container \"48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62\": container with ID starting with 48602cbe37ef38c40ff5228c972f634639acf618bcb1b257ffc2b26006ad6d62 not found: ID does not exist"
Sep 11 15:20:48.565313 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.565288    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"]
Sep 11 15:20:48.569130 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:48.569109    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-r6wbg"]
Sep 11 15:20:50.180243 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:50.180210    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cca1259-bec7-4714-95fe-7157e3767fa8" path="/var/lib/kubelet/pods/7cca1259-bec7-4714-95fe-7157e3767fa8/volumes"
Sep 11 15:20:54.620302 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:54.620267    2559 scope.go:117] "RemoveContainer" containerID="bde6d588dcef867901e6759d9fa3bb84a6353dc1496031785e54d0aa7f2a2fbc"
Sep 11 15:20:55.275417 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:20:55.275375    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:21:03.462776 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:03.462740    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:21:08.462654 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:08.462613    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-hzcgf"]
Sep 11 15:21:11.359634 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:11.359594    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-948bd5744-xs867_dba830e1-6091-4587-b619-92268c5bac62/authorino/0.log"
Sep 11 15:21:18.178356 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:18.178323    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-28tv4_165c4ad1-6229-42fc-a0dc-36db6cac1ffb/copy-manifests/0.log"
Sep 11 15:21:18.184647 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:18.184623    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-28tv4_165c4ad1-6229-42fc-a0dc-36db6cac1ffb/manager/0.log"
Sep 11 15:21:18.819154 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:18.819123    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7879c6fd8-czqxz_06536d2f-4a2f-4eca-a2e7-f88bf17cc92c/maas-api/0.log"
Sep 11 15:21:20.156600 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:20.156564    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-948bd5744-xs867_dba830e1-6091-4587-b619-92268c5bac62/authorino/0.log"
Sep 11 15:21:20.824425 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:20.824397    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-hzcgf_6aaec257-03ee-4ee4-9490-8305bf1ac755/limitador/0.log"
Sep 11 15:21:21.792537 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:21.792502    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789999dc4-tcljc_36b3ac8f-8c9a-4b52-8252-ce369f6f1e18/kube-auth-proxy/0.log"
Sep 11 15:21:22.829195 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:22.829162    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-njbrp_f8e3d64b-3965-4d96-ba52-6485f4d1ef9d/main/0.log"
Sep 11 15:21:23.058445 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:23.058414    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-2wtcx_62ab935e-838e-4597-ac0b-276d522ddbc8/main/0.log"
Sep 11 15:21:23.437719 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:23.437691    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-xtz29_e147bc5d-67c5-4713-8904-b7dcd06f3563/main/0.log"
Sep 11 15:21:29.879824 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:29.879788    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:21:31.628138 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:31.628104    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:21:33.069165 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:33.069135    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-24976_c89e9a20-c4e4-4c03-9601-8e47f2ad2df2/global-pull-secret-syncer/0.log"
Sep 11 15:21:33.201880 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:33.201843    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log"
Sep 11 15:21:33.309171 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:33.309089    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-223.ec2.internal_b26ed7e8c79abb7d6bd735b73ba0b5d9/haproxy/0.log"
Sep 11 15:21:37.709211 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:37.709127    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-948bd5744-xs867_dba830e1-6091-4587-b619-92268c5bac62/authorino/0.log"
Sep 11 15:21:37.865151 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:37.865126    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-hzcgf_6aaec257-03ee-4ee4-9490-8305bf1ac755/limitador/0.log"
Sep 11 15:21:39.567177 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:39.567145    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-cp67f_09522222-5124-4c7e-a196-39d46deeaa2e/cluster-monitoring-operator/0.log"
Sep 11 15:21:39.599790 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:39.599762    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-v2f5g_fd829058-20c0-48f3-b69e-d609e69ad9a9/kube-state-metrics/0.log"
Sep 11 15:21:39.622445 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:39.622415    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-v2f5g_fd829058-20c0-48f3-b69e-d609e69ad9a9/kube-rbac-proxy-main/0.log"
Sep 11 15:21:39.643165 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:39.643135    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-v2f5g_fd829058-20c0-48f3-b69e-d609e69ad9a9/kube-rbac-proxy-self/0.log"
Sep 11 15:21:39.718269 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:39.718187    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-666b8_4d344e80-b4d3-4ac9-9454-f1df618ba028/node-exporter/0.log"
Sep 11 15:21:39.737418 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:39.737388    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-666b8_4d344e80-b4d3-4ac9-9454-f1df618ba028/kube-rbac-proxy/0.log"
Sep 11 15:21:39.757531 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:39.757500    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-666b8_4d344e80-b4d3-4ac9-9454-f1df618ba028/init-textfile/0.log"
Sep 11 15:21:39.931269 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:39.931236    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-vksz9_574d7d83-4358-4a87-9500-0cc09631452f/kube-rbac-proxy-main/0.log"
Sep 11 15:21:39.955234 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:39.955183    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-vksz9_574d7d83-4358-4a87-9500-0cc09631452f/kube-rbac-proxy-self/0.log"
Sep 11 15:21:39.977484 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:39.977446    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-vksz9_574d7d83-4358-4a87-9500-0cc09631452f/openshift-state-metrics/0.log"
Sep 11 15:21:40.024764 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:40.024732    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe9ae9f8-afe3-43ee-855e-ca4181316d21/prometheus/0.log"
Sep 11 15:21:40.042785 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:40.042758    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe9ae9f8-afe3-43ee-855e-ca4181316d21/config-reloader/0.log"
Sep 11 15:21:40.061760 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:40.061731    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe9ae9f8-afe3-43ee-855e-ca4181316d21/thanos-sidecar/0.log"
Sep 11 15:21:40.080093 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:40.080068    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe9ae9f8-afe3-43ee-855e-ca4181316d21/kube-rbac-proxy-web/0.log"
Sep 11 15:21:40.100496 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:40.100469    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe9ae9f8-afe3-43ee-855e-ca4181316d21/kube-rbac-proxy/0.log"
Sep 11 15:21:40.122361 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:40.122332    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe9ae9f8-afe3-43ee-855e-ca4181316d21/kube-rbac-proxy-thanos/0.log"
Sep 11 15:21:40.141295 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:40.141267    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe9ae9f8-afe3-43ee-855e-ca4181316d21/init-config-reloader/0.log"
Sep 11 15:21:40.214075 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:40.214044    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-qf6ft_0b1f5560-3141-4801-98b6-52578799f5ed/prometheus-operator-admission-webhook/0.log"
Sep 11 15:21:40.248551 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:40.248525    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74849c8dd-gb62q_786b6811-09b6-4f34-93b3-a545605c0817/telemeter-client/0.log"
Sep 11 15:21:40.269521 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:40.269427    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74849c8dd-gb62q_786b6811-09b6-4f34-93b3-a545605c0817/reload/0.log"
Sep 11 15:21:40.290796 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:40.290761    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74849c8dd-gb62q_786b6811-09b6-4f34-93b3-a545605c0817/kube-rbac-proxy/0.log"
Sep 11 15:21:43.150616 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.150583    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-596d89868f-n4j2s_beac3c4c-6e77-4706-99cc-818b3d6d44e3/console/0.log"
Sep 11 15:21:43.263523 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.263488    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"]
Sep 11 15:21:43.263909 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.263891    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cca1259-bec7-4714-95fe-7157e3767fa8" containerName="maas-api"
Sep 11 15:21:43.264001 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.263912    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cca1259-bec7-4714-95fe-7157e3767fa8" containerName="maas-api"
Sep 11 15:21:43.264068 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.264034    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cca1259-bec7-4714-95fe-7157e3767fa8" containerName="maas-api"
Sep 11 15:21:43.267340 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.267317    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.269975 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.269955    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slnpk\"/\"openshift-service-ca.crt\""
Sep 11 15:21:43.271022 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.271001    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slnpk\"/\"kube-root-ca.crt\""
Sep 11 15:21:43.271141 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.271028    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-slnpk\"/\"default-dockercfg-w6qf6\""
Sep 11 15:21:43.275533 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.275506    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"]
Sep 11 15:21:43.361985 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.361939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-lib-modules\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.361985 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.361992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkr86\" (UniqueName: \"kubernetes.io/projected/58178eb5-e606-4f52-b5a2-17b40d2f2d97-kube-api-access-pkr86\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.362216 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.362058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-proc\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.362216 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.362105    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-sys\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.362216 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.362129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-podres\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.462733 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.462633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-lib-modules\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.462733 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.462685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkr86\" (UniqueName: \"kubernetes.io/projected/58178eb5-e606-4f52-b5a2-17b40d2f2d97-kube-api-access-pkr86\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.462733 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.462719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-proc\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.462973 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.462745    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-sys\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.462973 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.462765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-podres\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.462973 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.462836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-lib-modules\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.462973 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.462856    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-sys\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.462973 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.462836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-proc\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.462973 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.462912    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58178eb5-e606-4f52-b5a2-17b40d2f2d97-podres\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.471561 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.471532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkr86\" (UniqueName: \"kubernetes.io/projected/58178eb5-e606-4f52-b5a2-17b40d2f2d97-kube-api-access-pkr86\") pod \"perf-node-gather-daemonset-xxgf2\" (UID: \"58178eb5-e606-4f52-b5a2-17b40d2f2d97\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.578150 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.578105    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:43.849816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:43.849787    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"]
Sep 11 15:21:43.851762 ip-10-0-128-223 kubenswrapper[2559]: W0911 15:21:43.851718    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod58178eb5_e606_4f52_b5a2_17b40d2f2d97.slice/crio-64be466caa951a51bd3c0be01f87b2a3a1521867a57c8504471387be26efbace WatchSource:0}: Error finding container 64be466caa951a51bd3c0be01f87b2a3a1521867a57c8504471387be26efbace: Status 404 returned error can't find the container with id 64be466caa951a51bd3c0be01f87b2a3a1521867a57c8504471387be26efbace
Sep 11 15:21:44.815604 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:44.815561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2" event={"ID":"58178eb5-e606-4f52-b5a2-17b40d2f2d97","Type":"ContainerStarted","Data":"b7b234969709de6c5c92ca5cdf921d90caf959ff6a4fa4113e1f76f5f1c82490"}
Sep 11 15:21:44.816132 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:44.815653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2" event={"ID":"58178eb5-e606-4f52-b5a2-17b40d2f2d97","Type":"ContainerStarted","Data":"64be466caa951a51bd3c0be01f87b2a3a1521867a57c8504471387be26efbace"}
Sep 11 15:21:44.816132 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:44.815680    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:44.832769 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:44.832719    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2" podStartSLOduration=1.832699028 podStartE2EDuration="1.832699028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:44.830680172 +0000 UTC m=+2511.153308540" watchObservedRunningTime="2026-09-11 15:21:44.832699028 +0000 UTC m=+2511.155327397"
Sep 11 15:21:45.076223 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:45.076146    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w7klw_ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c/dns/0.log"
Sep 11 15:21:45.095776 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:45.095754    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w7klw_ecbd73b9-8f42-4afd-aa4d-38ae7f29ae9c/kube-rbac-proxy/0.log"
Sep 11 15:21:45.155713 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:45.155684    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7lxrw_7b7d0ed0-8337-4b8c-adfa-4918f5150593/dns-node-resolver/0.log"
Sep 11 15:21:45.799780 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:45.799746    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6r78k_c9533b2d-a936-407d-ad6f-e3480bcdd109/node-ca/0.log"
Sep 11 15:21:47.558333 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:47.558258    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789999dc4-tcljc_36b3ac8f-8c9a-4b52-8252-ce369f6f1e18/kube-auth-proxy/0.log"
Sep 11 15:21:48.620243 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:48.620210    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pqq7j_1ead5d7d-37b1-4bd4-a2c9-33d33d1a6238/serve-healthcheck-canary/0.log"
Sep 11 15:21:49.066831 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:49.066804    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-ctd5d_b35287e9-9950-4b9d-af5c-969dd954fead/insights-operator/0.log"
Sep 11 15:21:49.067535 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:49.067510    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-ctd5d_b35287e9-9950-4b9d-af5c-969dd954fead/insights-operator/1.log"
Sep 11 15:21:49.086962 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:49.086933    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6b6dq_2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf/kube-rbac-proxy/0.log"
Sep 11 15:21:49.106225 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:49.106199    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6b6dq_2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf/exporter/0.log"
Sep 11 15:21:49.125598 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:49.125565    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6b6dq_2b5e618a-1e46-4078-a3b2-8c2f97c4b8bf/extractor/0.log"
Sep 11 15:21:50.832083 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:50.832048    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-xxgf2"
Sep 11 15:21:52.084674 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:52.084642    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-28tv4_165c4ad1-6229-42fc-a0dc-36db6cac1ffb/manager/0.log"
Sep 11 15:21:52.103471 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:52.103432    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-28tv4_165c4ad1-6229-42fc-a0dc-36db6cac1ffb/copy-manifests/0.log"
Sep 11 15:21:52.798199 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:52.798163    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rm7gp_84d7fdb2-524e-4627-8740-17ba2dcfefa0/konnectivity-agent/0.log" limit=1024
Sep 11 15:21:53.541088 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:53.541062    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c79f8858d-crmtv_23913e6b-0ce6-4e0a-bd7a-924ea30c5b65/manager/0.log"
Sep 11 15:21:53.841152 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:53.841080    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789999dc4-tcljc_36b3ac8f-8c9a-4b52-8252-ce369f6f1e18/kube-auth-proxy/0.log"
Sep 11 15:21:56.364373 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:56.364289    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789999dc4-tcljc_36b3ac8f-8c9a-4b52-8252-ce369f6f1e18/kube-auth-proxy/0.log"
Sep 11 15:21:59.218315 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:59.218282    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-hkdtk_9c38c07c-2f0c-4e2a-a730-1280542e9fdc/kube-storage-version-migrator-operator/1.log"
Sep 11 15:21:59.219295 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:21:59.219274    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-hkdtk_9c38c07c-2f0c-4e2a-a730-1280542e9fdc/kube-storage-version-migrator-operator/0.log"
Sep 11 15:22:00.218375 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:00.218348    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2gmn2_a8003a60-3e02-4e75-b168-c540a3975f4d/kube-multus/0.log"
Sep 11 15:22:00.436952 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:00.436924    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r5kp_6bb6b12f-ef56-49af-a24e-9759831b31b7/kube-multus-additional-cni-plugins/0.log"
Sep 11 15:22:00.455622 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:00.455596    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r5kp_6bb6b12f-ef56-49af-a24e-9759831b31b7/egress-router-binary-copy/0.log"
Sep 11 15:22:00.477074 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:00.477052    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r5kp_6bb6b12f-ef56-49af-a24e-9759831b31b7/cni-plugins/0.log"
Sep 11 15:22:00.494816 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:00.494793    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r5kp_6bb6b12f-ef56-49af-a24e-9759831b31b7/bond-cni-plugin/0.log"
Sep 11 15:22:00.512647 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:00.512628    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r5kp_6bb6b12f-ef56-49af-a24e-9759831b31b7/routeoverride-cni/0.log"
Sep 11 15:22:00.532635 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:00.532616    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r5kp_6bb6b12f-ef56-49af-a24e-9759831b31b7/whereabouts-cni-bincopy/0.log"
Sep 11 15:22:00.549473 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:00.549432    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r5kp_6bb6b12f-ef56-49af-a24e-9759831b31b7/whereabouts-cni/0.log"
Sep 11 15:22:00.756518 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:00.756430    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2snh4_fde006ea-2cba-4b23-b83e-6f7703193a31/network-metrics-daemon/0.log"
Sep 11 15:22:00.771082 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:00.771059    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2snh4_fde006ea-2cba-4b23-b83e-6f7703193a31/kube-rbac-proxy/0.log"
Sep 11 15:22:01.701173 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:01.701147    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d6xtk_dc23afba-cb1c-4664-badc-e76289c18e6d/ovn-controller/0.log"
Sep 11 15:22:01.715614 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:01.715583    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d6xtk_dc23afba-cb1c-4664-badc-e76289c18e6d/ovn-acl-logging/0.log"
Sep 11 15:22:01.737138 ip-10-0-128-223 kubenswrapper[2559]: I0911 15:22:01.737112    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d6xtk_dc23afba-cb1c-4664-badc-e76289c18e6d/kube-rbac-proxy-node/0.log"